speculative execution

Running some benchmarks of hadoop using teragen/terasort. One of the recommendations I was given was to disable speculative execution. Noticed something rather strange when I forced it to disabled in the config.

Runtime with speculative execution: 18.5 minutes
Runtime without speculative execution: 1 hour

Seems that 2-3 map tasks are taking longer than the rest.

Question now is: why. Each map task is responsible for generating the same % of data - why would speculative execution make the job run quicker. Does this point to hardware differences ( if so, the slow tasks are on different machines - I have not noticed a pattern yet ), configuration problems elsewhere, or just random bad luck.

it is bottled!

Now a few weeks settling for carbonation.

2 full cases plus 3 bottles, not a bad yield.

The final gravity is 1.020 so it did not ferment off much more. Thats good, that just means the fermentable sugars are pretty much all digested. Its a little sweet but not too bad. Very smooth, a little oaky ( go fig! ) and you can really tell the difference with the Glenlivet. I think this recipe is done - probably will not change it.

Need to archive off a few bottles plus keep from giving away too much of this one. Speak up now for forever hold your peace!

cd /pub; more beer

A short while ago I picked up a really great book from the local homebrew supply shop. Extreme brewing ( http://store.dogfish.com/item/Extreme_Brewing/796/for_your_reading_pleasure/40/index.htm) by Sam Calagione over at dogfish head. I picked this book up for a few reasons:

1) It was the first homebrew book I had seen which discussed the use of oak chips and beer
2) The book has recipes to some of my favorite beers ( no, not just ones from Dog Fish Head! )
3) It has the recipe for Midas Touch :-> ( http://en.wikipedia.org/wiki/Midas_Touch_Golden_Elixir )

I got an idea from a few different sources:
1) The local brewery Barleys, they make a beer called Jack Frost which is a winter ale aged in a Jack Daniels cask
2) There is a recipe in the Extreme Brewing book which discusses using oak chips soaked in Port
3) I really love whisky!

Here is the idea:

Take a basic recipe - in this case the nut brown ale recipe from Dog Fish Head. Then take oak chips soaked in some kind of whisky - add to the secondary fermenter for a while. I tried this idea out 8 months ago or so with Jack Daniels whisky. I must say, it was one of my best brews to date. All of my friends and family ask me: When are you making that whisky beer again?

Well, Here we go. It has been brewed!

The recipe ( slightly modified from the original )

2lbs 2-row barley
10oz British amber malt
10oz chocolate malt
2oz roasted barley
2 cans light lme ( Alexanders I believe )
2lbs dark amber honey
1/2 oz warrior hops for the boil
1oz vanguard hops, last 15 minutes

I toasted some oak chips. Very simple, covered a cookie pan with aluminum foil, dropped the chips onto the tray and then cover with aluminum foil. Cook over a flame for about 15 minutes and then leave to cool. Some of the chips were pretty charred, some were not - which is ok that is what I want. Once the chips cooled, I tossed them into an air-tight canister along with 2oz or so of Glenlivet 12-year whisky ( I know, to some this is alcohol abuse but hey, its got a purpose! )




The recipe called for ringwood yeast but the local brew supply did not have this strain. I picked up the Wyeast 1028 London yeast which should be fairly close - we'll see.

Upon racking, the beer had a pretty strong foam on top. I assume this to be the protein rich brew. I awoke the next morning to the wonderful aroma of the digestion gasses of the yeast. The wife hates it - personally I love it :-> Beer had a nice rocky head and the airlock was going bonkers.





The steps still remaining for this batch:

1) Transfer to the secondary fermenter
2) Add the oak chips ( drain the whisky off, should cut down on sediment )
3) Leave in the secondary for at least 2 months - maybe more

I think the really long stay in the secondary should help this batch ferment out fully. The original gravity was 1.080 or so, but the temp may have been a little higher than the hydrometer accounted for. The difference is probably almost nothing.

the quest for web security

For a while now, I've often wondered just how secure my home stuff is. One area I've been wanting to tinker with is web traffic. The idea is, how difficult would it be to find and secure at least some of the content.

Idea 1: DNS

There are plenty of ways to prevent malware and adverts. If the dns resolves to localhost, or, say - your own internal web server for logging purposes... Kinda cool, but it seems as though this is not really being maintained anymore - not sure why, will have to research this one more.

Idea 2: Firefox plugins

NoScript is really cool. NoScript in combination with adblock and a few list subscriptions is fairly effective.

... todo, more content ...

Idea 3: Content filtering

For a while now I've protected my external apache instances with mod_security. I wondered though - could I apply mod_security within a virtual server and mod_proxy in combination?

If interested, this is how you could do it:

---- begin config block ----
LoadFile /usr/lib/libxml2.so.2
LoadModule security2_module modules/mod_security2.so
LoadModule unique_id_module modules/mod_unique_id.so
Include rules.d/modsecurity_crs_10_config.conf

Listen 3129

#ServerAdmin webmaster@dummy-host.example.com
DocumentRoot "/var/www/html"
ServerName someServer.somewhere
ErrorLog logs/proxy-error_log
CustomLog logs/proxy-access_log common

ProxyRequests On
ProxyVia On


Order deny,allow
Deny from all
Allow from 192.168.1.0/24

# Example configuration file for the mod_security Apache module

# This is the ModSecurity Core Rules Set.

Include rules.d/modsecurity_crs_20_protocol_violations.conf
Include rules.d/modsecurity_crs_21_protocol_anomalies.conf
Include rules.d/modsecurity_crs_23_request_limits.conf
Include rules.d/modsecurity_crs_30_http_policy.conf
Include rules.d/modsecurity_crs_35_bad_robots.conf
Include rules.d/modsecurity_crs_40_generic_attacks.conf
Include rules.d/modsecurity_crs_45_trojans.conf
Include rules.d/modsecurity_crs_50_outbound.conf

#Include rules.d/optional_rules/modsecurity_crs_20_protocol_violations.conf
Include rules.d/optional_rules/modsecurity_crs_40_generic_attacks.conf
#Include rules.d/optional_rules/modsecurity_crs_42_tight_security.conf
#Include rules.d/optional_rules/modsecurity_crs_21_protocol_anomalies.conf
Include rules.d/optional_rules/modsecurity_crs_42_comment_spam.conf
Include rules.d/optional_rules/modsecurity_crs_55_marketing.conf



---- end config block ----

Well, yes - it works quite nicely. The funny thing is, I started blocking all kinds of traffic. I thought it interesting to play around with their console application so I went to download it.

What do I notice while loading the page?

==> proxy-access_log <==
192.168.1.103 - - [29/Jan/2009:22:50:08 -0500] "CONNECT bsn.breach.com:443 HTTP/1.1" 200 -

==> modsec_audit.log <==
--fecc3827-A--
[29/Jan/2009:22:50:14 --0500] GeYs9MCoAZIAADzjiu0AAAAJ 192.168.1.103 53990 192.168.1.146 3129
--fecc3827-B--
CONNECT bsn.breach.com:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b2) Gecko/20081201 Firefox/3.1b2 Ubiquity/0.1.5
Proxy-Connection: keep-alive
Host: bsn.breach.com

--fecc3827-F--
HTTP/1.1 200 OK

--fecc3827-H--
Message: Warning. Match of "rx ^OPTIONS$" against "REQUEST_METHOD" required. [file "/etc/httpd/rules.d/modsecurity_crs_21_protocol_anomalies.conf"] [line "41"] [id "960015"] [msg "Request Missing an Accept Header"] [severity "CRITICAL"] [tag "PROTOCOL_VIOLATION/MISSING_HEADER"]
Message: Warning. Match of "rx ^((?:(?:POS|GE)T|OPTIONS|HEAD))$" against "REQUEST_METHOD" required. [file "/etc/httpd/rules.d/modsecurity_crs_30_http_policy.conf"] [line "37"] [id "960032"] [msg "Method is not allowed by policy"] [severity "CRITICAL"] [tag "POLICY/METHOD_NOT_ALLOWED"]
Apache-Handler: proxy-server
Stopwatch: 1233287408659700 5664183 (266 3214 -)
Producer: ModSecurity for Apache/2.5.2 (http://www.modsecurity.org/); core ruleset/1.6.0.
Server: Apache/2.2.3 (CentOS)

--fecc3827-K--
SecRule "&REQUEST_HEADERS:Accept" "@eq 0" "phase:2,chain,skip:1,log,auditlog,msg:'Request Missing an Accept Header',severity:2,id:960015,tag:PROTOCOL_VIOLATION/MISSING_HEADER"
SecRule "REQUEST_METHOD" "!@rx ^OPTIONS$" "phase:2,log,pass,t:none"
SecRule "&REQUEST_HEADERS:Content-Type" "@eq 0" "phase:2,pass,chain,log,auditlog,msg:'Request Containing Content, but Missing Content-Type header',id:960904,severity:4"
SecRule "REQUEST_METHOD" "!@rx ^((?:(?:POS|GE)T|OPTIONS|HEAD))$" "pass,status:501,phase:2,log,auditlog,msg:'Method is not allowed by policy',severity:2,id:960032,tag:POLICY/METHOD_NOT_ALLOWED"

--fecc3827-Z--

Yep, the mod_security ruleset is cutting some of the traffic from Breach Security - the makers of mod_security :-> Too funny!

It also handily blocks some other stuff. Since I configured my mac to use the proxy, it blocks apple's auth servers, mobile me ( which I am tinkering with as well - interesting but probably won't stick with it ), a whole bunch of odd stuff while downloading web pages and a few other things.

Oh, and if you throw in the in-memory caching on the proxy content the over-head of the mod_security is very nicely realized. I noticed response/load times improved a good bit.

This goes into the virtual server def:


CacheEnable mem /
MCacheSize 1024
MCacheMaxObjectCount 100
MCacheMinObjectSize 1
MCacheMaxObjectSize 2048


Funny thing, I had to turn off the proxy so I could publish the post. Guess I won't be leaving this on until I can debug it more. Still pretty cool!

if at first you don't succede

I have a semi-interesting post regarding a brew I came up with a while back. I modeled the batch off of a really nice beer - Tommy Knocker Maple beer ( http://tommyknocker.com/ourBeer.html#mapleNut )

I brewed this beer ... dunno, I know it has been 1.5 years because I brewed it about the time I moved into my current apartment - pretty much right when we moved in. When I was adding in the maple I accidentally added too much. The recipe called for 1 or 2 cups, I put in about double :-> Oops!

Well, when I fermented it it was nice. You could really smell the maple. When I bottled it and sampled the first one a few weeks later, it was pretty much un-drinkable. The maple was just so over-powering. It did not sour or get spoiled - it just did not taste good :-> I mentioned this to a friend of mine and his comment was rather fun! "Ah, just throw it in a case and toss it into the closet for a year. It'll mellow out.

Well, sure enough it has mellowed out nicely. Actually tastes ok ( though a tad sweet ).

Cheers! viva la laziness!

Further down the dark path

So, in May I picked up a MacBook Pro. I have now added to my apple collection :-> I picked up an iPhone 3G.

Current thoughts - well, so far I am pretty happy with it. I find the phone to be fairly well organized. Making use of the iCal, email remotely, web etc is just great. The mapping feature is quite nice - though I have found that google maps still does not have *everything* on there, but that is ok.

When I started goofing around with the phone I discovered the VPN support. I had honestly not noticed the fact that the phone has built-in support for VPN access. It supports pptp, l2tp and - of course the Cisco VPN. My new employer uses a Cisco VPN solution so I got all happy and fired it up. Works great - was very simple to configure.

It was actually quite a bit of fun freaking out my co-workers. I started loading up an internal-only web site on my phone and they started to freak out :-> When I explained to them just what was going on - the response was akin to the "ok, I gotta have a phone that can do the vpn stuff - that is just great".

All in all, the iPhone has been good. I've had it for just over a week now. I shelled out the extra cash for the 16gig model which I am happy I did. The 16gig, imho, is probably skimpy - but given the costs associated with making the phone 32gb I can understand the 8/16 models for now.

The phone design UI/interface is quite cool. Voicemail ad dialing are all menu-driven from the phone's screen. Adding 3-way calling is much more straightforward than my old cell phone.

There are a few things which I would like to see on the iPhone. It may support it and I just have not found out how to do it yet:

1) The ability to select/copy/paste text.
- and to then 'right click' or whatever and say search this or find this address etc
2) Disk access to the phone. Make the ability to create a space on the phone to store computer data and access it directly.
3) auto sync of things like wireless configuration settings to/from your primary machine. They support it for email/ical/address book, why not wireless network settings?
4) notes synchronization. I tend to use the stickies/notes for ... well, yea - notes! I would like the ability to sync between the phone/laptop. To me, the phone ( because it is a pda ) is a portable extension to my primary machine - the laptop. Maybe I can do this with a web-based application but I want them to remain private/non-web. EG, I won't always have net access and I want to retain access.

More later

its about time

Sometimes it is fun to gloat :-> Made the final payment on the XTerra today. About fucking time.

It has been through a lot and has served me well. Will be keeping it of course - no sense in ending a car payment early only to start another one!

This brings to mind an interesting rant I've had in the past. People's obsession with gas guzzling SUV's and their constant complaint that people should drive less. Think about it for a minute - if everyone who currently owns an SUV suddenly decides to sell it off - what is going to happen? Thats right, someone will buy it and re-sell it. That vehicle is going to stay on the road for some time. Yea its great to get more fuel efficient vehicles on the road - and I'm totally behind it. But seriously, its not going to solve much beyond: a) Filling the pockets of whatever auto manufacturer , b) filling the pockets of the bank whom you financed through ( IF you finance - ... hrm ) c) eat into your finances again.

There is this concept of cost/benefit and total cost of ownership. If you have a car payment of 500 a month and then add in gas at $4/gallon - what is your monthly cost?

a) 500 + ( (4*18)*3 ) + 150 = 866 per month
( $4 per gallon, 18 gallon fill-up. 3 fill-ups a month - then 150 for insurance )
b) 0 + ( (4*18)*3 ) + 150 = 366

Naturally, that is not including maintenance costs and such. We'll assume a few hundred over 3 months - it would average out. But wait, what happens if we get a cheaper car?

a) 350 + ( (4*10)*2) + 150 = 580
( Assume I get better mileage, 4-banger and all... fewer fill-ups etc )
b) 0 + ( (4*18)*3 ) + 150 = 366

That is of course assuming I can get an auto loan for $350/month. A lot of that depends on what kind of car I get and how much I put down - I fully admit. Working with simple numbers here. ( What I really want is a hybrid - but that puts me back at the freakin 400-500/month payment! )