tag:blogger.com,1999:blog-70421477555945949262024-03-13T10:38:00.335-07:00roggrA blog about technology, Web startups, and other things.rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.comBlogger71125tag:blogger.com,1999:blog-7042147755594594926.post-81043567977815997682021-03-26T21:57:00.007-07:002021-03-26T21:58:43.307-07:00Docker and Python debugging in VS Code error: timed out waiting for launcher to connect<p>Scenario: you have a Linux machine and you're creating a Dockerized Python app. You fire up Visual Studio Code, follow <a href="https://code.visualstudio.com/docs/containers/quickstart-python" rel="nofollow" target="_blank">the official instructions</a>, and eagerly hit F5 to test your app.</p><p>But nothing happens, except an error dialog after about <a href="https://github.com/microsoft/debugpy/pull/313/files#diff-2e163a29eeeb679e663816e49309b606feb255df83d4aae6eb4784c164adbba7R13" rel="nofollow" target="_blank">15 seconds</a>:</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6yhOjEQEpTopKIThBZjsk_hZW6-j7uBsbseyg0z38hHkF3Pf_q-TBm9uk7DXZhsnDDqVSsXkswAtB_N8Ifcjd2IW7e3x3ukx_LqHpPwAiUzVMPWwsznwKhY6vTXiWcOlqYckDNkh7e7_2/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="175" data-original-width="528" height="106" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6yhOjEQEpTopKIThBZjsk_hZW6-j7uBsbseyg0z38hHkF3Pf_q-TBm9uk7DXZhsnDDqVSsXkswAtB_N8Ifcjd2IW7e3x3ukx_LqHpPwAiUzVMPWwsznwKhY6vTXiWcOlqYckDNkh7e7_2/" width="320" /></a></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">This may be your firewall blocking connections on the docker network interface.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">To confirm this, turn off your firewall, launch a debug process in VS Code, and see if the debug session starts. Repeat with the firewall on. If that fails, it's time to add rules allowing traffic for the docker interface.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">First, find the subnet for your docker interface:</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirqS6KfvCyRlScR2_VYVT3nUlUCKC5oXx6JaIzhDZo4YUhBVVuNMysh5ApVQSpnp9JK70_-BgS8x3dxIjI3_gNfTRnCOTgk4htJxUagkPurlDB7-7msLda3gWZSTLYohpPWs0KRNr5xe4j/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="141" data-original-width="729" height="62" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirqS6KfvCyRlScR2_VYVT3nUlUCKC5oXx6JaIzhDZo4YUhBVVuNMysh5ApVQSpnp9JK70_-BgS8x3dxIjI3_gNfTRnCOTgk4htJxUagkPurlDB7-7msLda3gWZSTLYohpPWs0KRNr5xe4j/" width="320" /></a></div><br /><br /></div><div class="separator" style="clear: both; text-align: left;">In this instance, you want traffic allowed through docker0 over its IP range, which in my case is 172.17.0.0/16. Yours may be different.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">In a terminal, add those rules:</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><span style="font-family: monospace;"><span style="background-color: white;">sudo ufw allow out on </span><span style="background-color: white; font-weight: bold;">docker</span><span style="background-color: white;">0 from 172.17.0.0/16
</span><br />sudo ufw allow in on <span style="background-color: white; font-weight: bold;">docker</span><span style="background-color: white;">0 from 172.17.0.0/16</span></span></div><div class="separator" style="clear: both; text-align: left;"><span style="font-family: monospace;"><span style="background-color: white;"><br /></span></span></div><div class="separator" style="clear: both; text-align: left;"><span style="background-color: white;"><span style="font-family: inherit;">Reload your firewall config, and your debugging should start working.</span></span></div><div class="separator" style="clear: both; text-align: left;"><span style="font-family: monospace;"><span style="background-color: white;"><br /></span></span></div><div class="separator" style="clear: both; text-align: left;"><span style="font-family: monospace;"><span style="background-color: white;">sudo ufw reload</span></span></div><div class="separator" style="clear: both; text-align: left;"><span style="font-family: monospace;"><span style="background-color: white;"><br /></span></span></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><span style="font-family: monospace;"><br />
<br /></span></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><br /></div><br /><br /><p></p>rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-82020856416115216372021-02-12T20:58:00.000-08:002021-02-12T20:58:09.092-08:00How to Resolve Windows Host Names on your LAN from a Linux Machine<p>Resolving Windows host names on a LAN from a Linux machine is tricky and changes often enough that most AskUbuntu or Stack Overflow "solutions" don't work as of Feb 2021.</p><p>This method works in KDE Neon 5.20 with Kernel 5.4.0-65-generic and should work with most Ubuntu 20.x versions.</p><p>The idea is to give your Ubuntu machine your router's / DHCP server's address as a secondary DNS server.</p><p>The difficulty is that the standard <span style="font-family: courier;">/etc/resolv.conf</span> file gets clobbered when the <span style="font-family: courier;">NetworkManager</span> service starts. The trick is how to manage <span style="font-family: courier;">resolv.conf</span> manually.</p><p>Prerequisites: </p><p></p><ul style="text-align: left;"><li>A LAN</li><li>A Windows host on the LAN with default network configuration, in WORKGROUP</li><li>A Ubuntu 20.x machine</li></ul><p></p><p>All of the following happens on the Ubuntu machine. The Windows host doesn't need any reconfiguration.</p><p>Steps:</p><p></p><ol style="text-align: left;"><li>Tell <span style="font-family: courier;">NetworkManager</span> not to provide DNS resolution</li><li>Create your own <span style="font-family: courier;">resolv.conf</span></li></ol><div><br /></div><div>Step 1 (<a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/manually-configuring-the-etc-resolv-conf-file_configuring-and-managing-networking" target="_blank">described here</a>)</div><div><div>Create <span style="font-family: courier;">/etc/NetworkManager/conf.d/90-dns-none.conf</span> file with the following content:</div><div><br /></div><div><span style="font-family: courier;">[main]</span></div><div><span style="font-family: courier;">dns=none</span></div></div><div><br /></div><div>Step 1.1</div><div>Restart NetworkManager</div><div><span style="font-family: courier;">sudo systemctl reload NetworkManager</span></div><div><br /></div><div>Step 2</div><div>If <span style="font-family: courier;">/etc/resolv.conf</span> is a symlink to a systemd file (typically <span style="font-family: courier;">/run/systemd/resolve/resolv.conf </span>or the stub file), remove that symlink.</div><div>Create a<span style="font-family: courier;">/etc/resolv.conf</span> file like this (replace 192.168.1.1 with your router or DHCP server's actual IP address if it's different):</div><div><br /></div><div><div><span style="font-family: courier;">nameserver 127.0.0.53</span></div><div><span style="font-family: courier;">nameserver 192.168.1.1</span></div></div><div><br /></div><div>Step 2.1 </div><div>Restart the resolver </div><div><div><span style="font-family: courier;"><br /></span></div><div><span style="font-family: courier;">sudo systemctl restart systemd-resolved</span></div></div><div><br /></div><div>Then you can test like this:</div><div><br /></div><div><div><span style="font-family: courier;">$ nslookup google.com</span></div><div><span style="font-family: courier;">Server: 127.0.0.53</span></div><div><span style="font-family: courier;">Address: 127.0.0.53#53</span></div><div><span style="font-family: courier;"><br /></span></div><div><span style="font-family: courier;">Non-authoritative answer:</span></div><div><span style="font-family: courier;">Name: google.com</span></div><div><span style="font-family: courier;">Address: 216.58.195.78</span></div><div><span style="font-family: courier;">Name: google.com</span></div><div><span style="font-family: courier;">Address: 2607:f8b0:4005:807::200e</span></div><div><span style="font-family: courier;"><br /></span></div><div><span style="font-family: courier;">$ nslookup mywindowsmachine</span></div><div><span style="font-family: courier;">;; Got SERVFAIL reply from 127.0.0.53, trying next server</span></div><div><span style="font-family: courier;">Server: 192.168.1.1</span></div><div><span style="font-family: courier;">Address: 192.168.1.1#53</span></div><div><span style="font-family: courier;"><br /></span></div><div><span style="font-family: courier;">Name: mywindowsmachine</span></div><div><span style="font-family: courier;">Address: 192.168.1.61</span></div><div><span style="font-family: courier;">;; Got SERVFAIL reply from 127.0.0.53, trying next server</span></div><div><br /></div></div><div><br /></div><div>If the above still doesn't work, try installing winbind. </div><div><br /></div><div><div><span style="font-family: courier;">sudo apt install winbind libnss-winbind</span></div><div><br /></div></div><p></p>rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-30209768277433870272018-12-09T17:06:00.001-08:002018-12-09T18:13:27.312-08:00Ergonomics at the computer: keyboards<h2>
Keyboards</h2>
As the primary method of entering data into a computer, keyboards play a significant role in a workstation's ergonomics and comfort.<br />
<br />
This is merely an outline, and I will fill it out over time.<br />
<br />
<h3>
Split Keyboards</h3>
<div>
Split keyboards come in two major flavors: all-in-one, or physically separate halves. The idea is to be able to position and orient each half of the keyboard so that each of your hands is in an optimally comfortable position.<br />
<br />
One-piece split keyboards tend to be raised in the middle and have their keys laid out in a "smile line" arrangement. The most famous of those is likely the <a href="https://www.blogger.com/u/2/blogger.g?blogID=7042147755594594926#msnatural">Microsoft Natural Keyboard</a>. Alternatively, some have keys in concave key wells, like the <a href="https://www.blogger.com/u/2/blogger.g?blogID=7042147755594594926#advantage">Kinesis Advantage</a> or <a href="https://www.maltron.com/store/p21/Maltron_L89_dual_hand_fully_ergonomic_%283D%29_keyboard_-_US_English.html" target="_blank">Maltron</a>. Then there's the <a href="https://www.blogger.com/u/2/blogger.g?blogID=7042147755594594926#safetype">Safetype</a>, which is really its own beast.<br />
<br />
The other family of multi-piece keyboards include the <a href="https://www.blogger.com/u/2/blogger.g?blogID=7042147755594594926#freestyle">Kinesis Freestyle</a>, <a href="https://www.blogger.com/u/2/blogger.g?blogID=7042147755594594926#vea">VE.A and clones</a>, and <a href="https://keeb.io/" target="_blank">a raft of others</a>.</div>
<h3>
Mechanical vs Membrane / Rubber Dome</h3>
<div>
<a href="https://www.reddit.com/r/MechanicalKeyboards/comments/8m79uo/its_prime_shitposting_time_boys/" target="_blank">Holy wars rage</a> over what feels / looks / sounds better of mechanical switches (e.g. Cherry MX, buckling springs, Alps, etc) vs membranes or rubber domes.<br />
<br />
Like everything else, each keyboard type has its pros and cons. There are some very high quality membrane keyboards with very low key actuation force, which is a great way to reduce stress on your muscles and tendons (viz. Kinesis Freestyle). On the other hand, mechanical switches come in a bewildering number of variations, allowing you to customize almost everything: key travel, actuation force, bump vs. no bump, audible click vs. not, pitch and loudness, even housing and stem colors. There's <a href="https://www.reddit.com/r/MechanicalKeyboards/" target="_blank">a cottage industry of geeky enthusiasts</a> who take switches apart and recombine the parts into hybrid creations designed to maximize one characteristic or another. It's a little crazy, but more power to them, and I've benefited from that kind of <a href="https://input.club/the-comparative-guide-to-mechanical-switches/" target="_blank">mad scientist approach</a> via some novel switches like <a href="https://www.reddit.com/user/hbheroinbob/" target="_blank">Outemu Sky </a>and Zealios, which arguably wouldn't exist if people were content with stock switches from Cherry, Gateron, Kailh, or Outemu.<br />
<br />
In some cases, the argument between mechanical or membrane is truly pointless: some keyboards only come in one style, usually membrane, so if you like those models, you have to deal with a membrane, period. This applies to the MS Natural keyboard, Gold Touch, Kinesis Maxim (all membrane); and on the other hand the Kinesis Advantage isn't available with membrane keys, which is a bummer if you like Kinesis's membrane options like the <a href="https://www.amazon.com/gp/product/B0016A0RLA/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=B0016A0RLA&linkCode=as2&tag=roggr-20&linkId=6278ff886a13207fb95279d016b16103" target="_blank">Kinesis Freestyle Solo</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=roggr-20&l=am2&o=1&a=B0016A0RLA" style="border: none !important; margin: 0px !important;" width="1" />.<br />
<br />
<a href="https://www.kinesis-ergo.com/" target="_blank">Kinesis</a> recently released two mechanical Freestyle models (the <a href="https://www.kinesis-ergo.com/shop/freestyle-pro/" target="_blank">Freestyle Pro</a> and the <a href="https://www.amazon.com/gp/product/B06XSPGL9S/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=B06XSPGL9S&linkCode=as2&tag=roggr-20&linkId=30b9bc37a6286321d2577a9e2ecb50bc" target="_blank">Freestyle Edge</a>), which I haven't tried yet, but it's one of those rare cases where you can actually choose between a membrane or a mechanical version of essentially the same keyboard.</div>
<h3>
Tenting and Tilting</h3>
<div>
Tenting refers to the ability to lift each half of a split keyboard such that the center is higher than the outer edges. The result is a keyboard that looks like an A-frame.<br />
<br />
Tilting refers to adding a front-to-back angle to a keyboard (split or not). Most keyboards sold today have built-in feet (usually retractable) that allow you to give the keyboard a "positive tilt" (the number row is taller than the space bar), even though that tilt is ergonomically questionable, as it makes your wrist angle more from a relaxed, straight position. Ergo aficionados often add "negative tilt" to their keyboards (the space bar is the tallest key, and the num row is lower), which helps keep your wrists on the same plane as your forearms, with no bending.<br />
<br />
The <a href="https://ultimatehackingkeyboard.com/start/feet" target="_blank">Ultimate Hacking Keyboard comes with feet and mounting holes</a> that let you choose what kind of tilt you want (if any), instead of forcing you to use positive or no tilt, like most keyboards do. </div>
<div>
<br /></div>
<h3>
Personal History</h3>
<div>
Here's an incomplete list of keyboards I've used over the years.</div>
<div>
<br /></div>
<h4>
Ultimate Hacking Keyboard, Cherry MX Clear</h4>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://ultimatehackingkeyboard.com/wordpress/wp-content/uploads/configurator_base_text_ansi_black_linux-500x200.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="200" data-original-width="500" height="80" src="https://ultimatehackingkeyboard.com/wordpress/wp-content/uploads/configurator_base_text_ansi_black_linux-500x200.png" width="200" /></a></div>
Great! I'll write a dedicated post soon. I just started using it, but it's the most promising keyboard I've tried in a while.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://tr1.cbsistatic.com/hub/i/2017/01/27/3bf913f2-c9f9-42bd-a855-60172cced860/fig-a-1-23.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="451" data-original-width="500" height="180" src="https://tr1.cbsistatic.com/hub/i/2017/01/27/3bf913f2-c9f9-42bd-a855-60172cced860/fig-a-1-23.png" width="200" /></a></div>
<h4>
<a href="https://www.blogger.com/u/2/null" name="msnatural"></a>Microsoft Natural Keyboard gen 1, 1994</h4>
Heavy-duty construction, heavy key presses, very good. I still use it on occasion.<br />
<h4>
</h4>
<h4>
</h4>
<h4>
Microsoft Natural Keyboard, early 2000s</h4>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://images-na.ssl-images-amazon.com/images/I/41TOc8nO2TL._SX425_.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="234" data-original-width="425" height="110" src="https://images-na.ssl-images-amazon.com/images/I/41TOc8nO2TL._SX425_.jpg" width="200" /></a></div>
The best era, hands down <span style="font-size: xx-small;">(see what I did there?)</span> No multimedia button bs, light key presses, about as good as a mainstream ergo keyboard gets. I'll never understand why Microsoft discontinued this model. I still use one on occasion.<br />
<h4>
</h4>
<h4>
</h4>
<h4>
Microsoft Natural Keyboard, mid 2000s (black)</h4>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://pisces.bbystatic.com/image2/BestBuy_US/images/products/7332/7332059_sd.jpg;maxHeight=640;maxWidth=550" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="332" data-original-width="550" height="120" src="https://pisces.bbystatic.com/image2/BestBuy_US/images/products/7332/7332059_sd.jpg;maxHeight=640;maxWidth=550" width="200" /></a></div>
Lots of useless media and other keys; feels pretty good, except the space bar, which is much too heavy. Most workplaces in Silicon Valley have one of those lying around, so I'll use it if it's my only ergo option, but the heavy space bar is problematic.<br />
<h4>
</h4>
<h4>
</h4>
<h4>
Kinesis Maxim</h4>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://www.fentek-ind.com/maxim.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="210" data-original-width="337" height="124" src="https://www.fentek-ind.com/maxim.jpg" width="200" /></a></div>
A funky split/tenting/splayed rubber dome. Works well, and has been my main keyboard at work for some time. I still use it daily.<br />
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
<a href="https://www.blogger.com/u/2/null" name="advantage"></a>Kinesis Advantage, Cherry MX Brown</h4>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://www.kinesis-ergo.com/wp-content/uploads/2016/09/kb600-angled-cc-510x344.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="344" data-original-width="510" height="134" src="https://www.kinesis-ergo.com/wp-content/uploads/2016/09/kb600-angled-cc-510x344.jpg" width="200" /></a></div>
Very good, though it feels a little chintzy and the learning curve is steep. Becoming truly fluent with it would require using it exclusively for a while, and that hasn't been realistic given my day-to-day. I still use it occasionally.<br />
<h4>
</h4>
<h4>
<a href="https://www.blogger.com/u/2/null" name="freestyle"></a>Kinesis Freestyle</h4>
<a href="https://www.kinesis-ergo.com/wp-content/uploads/2012/08/Kinesis-Freestyle2-for-PC-9.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="344" data-original-width="510" height="134" src="https://www.kinesis-ergo.com/wp-content/uploads/2012/08/Kinesis-Freestyle2-for-PC-9.jpg" width="200" /></a>This was my daily keyboard for quite some time. I made a tenting kit for it because Kinesis charges way too much for theirs. Very light keypresses, good layout. I still use it occasionally.<br />
<h4>
</h4>
<h4>
</h4>
<h4>
Gold Touch</h4>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://www.allthingsergo.com/wp-content/uploads/2011/11/Adjusting.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="480" data-original-width="608" height="157" src="https://www.allthingsergo.com/wp-content/uploads/2011/11/Adjusting.jpg" width="200" /></a></div>
This is a split with a handle to lock it into position. Not a great key feel, and that friction hinge thing feels like trouble down the line. I sold it.<br />
<br />
<br />
<br />
<br />
<br />
<br />
<a href="https://www.heatware.com/tmp/market_photos/files/medium/IMG_20170703_152357.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="600" data-original-width="800" height="149" src="https://www.heatware.com/tmp/market_photos/files/medium/IMG_20170703_152357.jpg" width="200" /></a>They also made a "travel" version (the Gold Touch Go)<br />
with cheap laptop keys and it was horrendous. Better than nothing, but not by much. I got rid of it too.<br />
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
Viterbi, Outemu Sky</h4>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://cdn.thingiverse.com/renders/df/27/24/00/b2/71ab607e9934c5880b45b767ed5b0a27_preview_featured.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="472" data-original-width="628" height="150" src="https://cdn.thingiverse.com/renders/df/27/24/00/b2/71ab607e9934c5880b45b767ed5b0a27_preview_featured.jpg" width="200" /></a></div>
My first foray into ortholinear keyboards. It doesn't feel terribly different from staggered layouts. I just started using it so I haven't decided how much I like it yet.<br />
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
Koolertron, Cherry MX Brown</h4>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://images-na.ssl-images-amazon.com/images/I/61ahra0jVDL._SX425_.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="425" data-original-width="425" height="200" src="https://images-na.ssl-images-amazon.com/images/I/61ahra0jVDL._SX425_.jpg" width="200" /></a></div>
Very good construction, fully programmable, good quality keycaps, and each half is usable on its own. The layout is a little odd with extra keys that probably shouldn't be there. I never really got attached to this one, though there's really nothing wrong with it.<br />
<br />
<h4>
</h4>
<h4>
</h4>
<h4>
<a href="https://www.blogger.com/u/2/null" name="safetype"></a>Safetype</h4>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://images-na.ssl-images-amazon.com/images/I/41LpxbFIxJL.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="354" data-original-width="500" height="141" src="https://images-na.ssl-images-amazon.com/images/I/41LpxbFIxJL.jpg" width="200" /></a></div>
This is an interesting beast. The alphas, mods, and function keys are vertical, while nav and numpad keys are in the center console, between the two upright halves. Some of the keycaps are printed backwards, and the keyboard has mirrors on each half so you can see what you're doing. I am not used to this keyboard yet, but it's a lot easier to use than I anticipated.<br />
<br />
<h4>
</h4>
<h4>
<a href="https://www.blogger.com/u/2/null" name="vea"></a>VE.A Clone, Gateron Brown</h4>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://i.redd.it/cfktmtqevc801.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="600" data-original-width="800" height="150" src="https://i.redd.it/cfktmtqevc801.jpg" width="200" /></a></div>
This is a clone of a very expensive Korean split keyboard. Mine has Gateron brown tactile switches. Very good, usable layout, funky backlighting, usable extra keys on the left side. I made a custom tenting kit for it, and use it daily.<br />
<br />rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-63878200485722836752018-12-09T16:42:00.003-08:002018-12-09T17:46:44.968-08:00Ergonomics at the computer, part 1<h2>
Background</h2>
I started typing a long, long time ago, in a <strike>galaxy</strike> country far far away. As an avid nerd, that meant programming or playing Tau Ceti for hours and hours every day. I didn't learn to type "properly" (that concept wasn't taught in schools at the time) and almost certainly developed terrible typing habits. Add to that the desire and ability to type pretty fast (95 wpm on average), and a couple of careers that relied on typing a lot, and it's not surprising I've been dealing with RSI (repetitive stress injury) for quite some time. This has led to a lifelong pursuit of ergonomics.<br />
<br />
While it's fun to try various kinds of keyboards, mice, and computing devices, in my case fun is only a side benefit. Using standard devices is not possible: typing on a straight or laptop keyboard for just a few minutes will trigger a tendinitis flare-up that can be partially or completely disabling for days. As someone who makes a living typing all day, not being able to type is problematic. So I've been trying a lot of devices and learned a few things that may be useful to my two <strike>million</strike> readers, hence this post.<br />
<h2>
Outline</h2>
The plan is to discuss various approaches and devices I've tried over the years, with some semblance of structure. Topics will include keyboards, mice, trackballs, touchpads, tablet devices (with and without styluses), and accessories like keyboard trays, mouse pads, wrist/palm rests, tenting kits, etc.<br />
<br />
<a href="https://www.roggr.com/2018/12/ergonomics-at-computer-keyboards.html" target="_blank">Part 1: Keyboards</a>rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-29572137842046262722017-01-31T20:08:00.001-08:002017-01-31T20:18:08.707-08:00The Planning DilemmaAfter a few years in the tech industry in Silicon Valley, I've noticed a recurring theme in growing startups: planning is a problem. Now this isn't a novel insight--<a href="https://www.amazon.com/Rework-Jason-Fried/dp/0307463745" target="_blank">books</a> have been written (partially) on the topic, successful <a href="https://pivotal.io/labs" target="_blank">consultancies</a> have been built (partially) to solve this problem, and lots of very smart people have been working on it for a long time.<br />
<br />
Ultimately, however, every organization needs to figure out what sets of practices work best for its own unique circumstances: how big and experienced their team is, what kind of product they're building, how risk-averse and regulated their sector or customers are, and many other considerations. Much like learning to read, dance or play the piano, it's up to the learner to put in the work so they can become good at it. Planning is no different.<br />
<h3>
The Growth of Planning</h3>
The particular dilemma that led me to write this post is a common trajectory I've seen in quite a few growing startups grappling with the question of "how do we plan". It tends to go like this:<br />
<br />
1. No planning. Feature / task prioritization is obvious, the number of people doing the work is small, and you don't have time to do anything but build as fast as you can.<br />
<br />
2. You have your beta out, enough to get a little funding (maybe a big seed or even an A round) so you can actually hire a couple of engineers, a product manager, maybe a designer, beyond the founder(s) and initial hire(s). You may have traffic to your app, or actual customers using your service, maybe a product-focused board member, a trade show or hard date you want to hit in a few months to demo your app to secure more funding. Whatever the reason, you start putting some planning in place, maybe with daily stand-ups to coordinate between teams, or weekly sprint planning. Still fairly lightweight. Your leads/execs get together once every 2-4 weeks for higher-level feature prioritization and roadmapping, more or less often depending on your product and industry.<br />
<br />
3. You've secured funding and hired more people. You now have 2-3 distinct teams working on separate features (web and mobile, Android and iOS, e-commerce and logistics, design and manufacturing, whatever). You now have a fully-fledged marketing team with "<a href="http://www.leadingresults.com/ramblings/are-your-pr-activities-rolling-thunder-or-just-a-firecracker" target="_blank">rolling thunder</a>" tactics, media tours to plan, conference appearances to organize, trade show booths to buy, ad spend budgets to propose, etc. You may have a manufacturing pipeline with very real lead times. The point is that <i>people outside of engineering need to know what's happening when,</i> sometimes pretty far in advance.<br />
<br />
Yes, we've that software estimates are lies, that "it'll ship when it's ready," and other enlightened developer-friendly manifestos. I won't discuss the merits of that point in this post, and simply stipulate someone somewhere needs to know what's happening when beyond the next few days.<br />
<h3>
Planning For Others</h3>
At this stage of growth, in most (all?) of the startups I've observed or been a part of, the typical setup from step 2 expands to:<br />
<br />
a) 2-3 teams, each with dedicated product, design and engineering resources; then<br />
<br />
b) management overhead to coordinate between those teams and make sure the company's overall goals and priorities are being reflected correctly; then<br />
<br />
c) meta-management overhead to make sure the meta-product management from b) is communicated effectively to people outside the product/engineering org (e.g. marketing, sales, execs, board, etc)<br />
<br />
In practice, that often looks like this:<br />
<br />
<ul>
<li>execs meet, often "off-site", once a quarter or so, to define the Big Goals or Roadmap or whatever it's called for the company; then</li>
<li>division heads from b) (VP product, VP engineering) have regular planning sessions to define and prioritize the next batch of work for some period (1-4 weeks) (the "pre-sprint planning")</li>
<li>each division head meets with "their" leads (PM, lead eng) from each team in a) to clarify and prioritize stories for the upcoming sprint (the "sprint planning", per team)</li>
<li>the division heads communicate any relevant timing updates to their peers (e.g. sales or marketing)</li>
<li>each team has their own internal touch points (daily standups are still common)</li>
<li>each team may elect to do a post-sprint retrospective to look back at the sprint, see what was good and what could be improved, to feed into the next sprint</li>
</ul>
<br />
Given that sprints tend to be brief in order to be able to respond quickly to changing priorities (sometimes as short as one week), and that any sufficiently meaty story needs a fair bit of time to write, read and understand before it can be implemented, I'd estimate the above planning duties combined make up at least <b>a whole day's work per week per developer, </b>and probably about three days' work per week per division head.<br />
<br />
If you want to check my math:<br />
<br />
For developers:<br />
<br />
<ul>
<li>sprint planning = 2-4 hours</li>
<li>retrospective = 2 hours</li>
<li>daily standup = 5 x 12 min = 1 hour</li>
<li>context switching costs = 1-2 hours over the course of the week</li>
<li>ancillary negotiation, hemming, hawing, clarification, etc. over stories = 1-2 hours</li>
</ul>
<br />
Twenty percent of a developer's time spent on process management in the name of agility seems excessive to me.<br />
<h3>
Process Artifacts</h3>
Anecdotally, both as a developer and as a manager, I have heard feedback that <b>daily stand-ups </b>as a regular event aren't terribly useful (they are a costly interruption / context switch and often devolve into status reports that aren't useful to anybody or tangents that take too long and engage far too many people who are too polite to just leave a conversation they're not involved in; dependencies and blockers are best dealt with one on one, not as a group).<br />
<br />
I have also participated in and/or led <b>retrospectives </b>and have mixed feelings about them; as blameless post-mortems (not necessarily after an outage: they could be held after a good or a bad/stressful release) I find them to be good vehicles for venting and constructive criticism that can lead to improvements, but as meta-exercises about the process, I find them self-serving and unnecessary.<br />
<br />
<b>Sprint planning </b>is tricky. Writing stories is difficult; estimating them arguably harder; story points, planning poker and other artifacts are questionable; <a href="https://www.joelonsoftware.com/2007/10/26/evidence-based-scheduling/" target="_blank">evidence-based scheduling</a> seems more promising, but I feel you need a dedicated technical but non-coding manager to deal with it reliably and dispassionately, and that's hard to come by in a small startup.<br />
<h3>
What's The Upshot?</h3>
I don't have a good answer. At this point I'm pretty familiar with the new software development orthodoxy (most things under the "agile" umbrella), and while I find it preferable to a waterfall approach in principle (a system that is gleefully mocked by many people who've never actually experienced it; I certainly haven't, and I've been around a while), I feel a lot of its standard assumptions, tools and practices need to be challenged objectively and dispassionately. Otherwise it's just bandwagoning, and it's the opposite of what it's supposed to be--pragmatic.<br />
<br />
I'd love to hear your thoughts and experience and learn from people who've been dealing with this. Please comment or get in touch!rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-58526553508611709172016-12-20T11:52:00.001-08:002016-12-20T11:52:31.954-08:00Fiio X5 high-def media player: quick reviewI recently upgraded an iPod Classic 160GB to a higher-definition, larger-capacity Fiio X5 (first generation). The main selling points for me were:<br />
<br />
<ul>
<li>dual micro-SD card slots for up to 512GB of storage </li>
<li>high-quality DAC</li>
<li>support for all the formats I use</li>
<li>good battery life</li>
<li>frequent firmware updates</li>
</ul>
<div>
Unfortunately, the device I got has to go back. It's probably defective, but it also has significant UI problems (even in firmware version 2.6). Here's a summary.</div>
<div>
<ul>
<li><b>turning the device on almost never works</b>. I always have to reset it (hold the power button down for 15+ seconds) and then try to turn it on (hold power button down for 2 seconds). Most of the time, that doesn't work and I have to do the whole operation again. Occasionally several times. This means it takes 1-2 minutes just to get the device ready to use. iPod = instant, and it always works.</li>
<li><b>the hardware buttons do not work when the display goes to sleep</b>. To turn the volume up or down, pause, play or skip, you have to click the power button, then the hardware button you want. </li>
<ul>
<li>Sure, you could set the display never to go to sleep, but you lose the battery-saving benefits. And the default behavior is for the display to go to sleep, which means the out-of-the-box configuration doesn't work the way you'd expect, and requires two clicks to perform any function.</li>
</ul>
<li>the two points above often combine for maximum annoyance. <b>When the display goes to sleep</b> (or the device enters some kind of low-power mode that lets the music play), <b>and the power button doesn't work</b>, which means <b>you can't pause / unpause / skip / adjust the volume</b>. The only option is to reset the device (hold power button down for 15+ seconds), start it up, try again if it didn't work, and make your change.</li>
<ul>
<li>Except <b>when the device comes back from a reset, it doesn't remember what track was playing</b>, so you have to browse your library all over again, find whatever you were playing, and play it again.</li>
</ul>
<li>browsing the SD card or the library is impossibly slow. </li>
<ul>
<li>The jog wheel scrolls through the library at the same speed, no matter how quickly you're jogging. On the iPod, after a certain speed, the scrolling speeds up and skips whole letters in the alphabet all at once. On the Fiio X5, if you have 500 artists and want to listen to Zimmer's Hole or Zoe Keating, you'd better have a sandwich at hand, because it's going to take a while.</li>
<li>When you're tired of Zimmer's Hole and want to switch to Frank Zappa, each step (tracks -> album -> artist (Zimmer's Hole) -> scroll -> artist (FZ) -> album -> tracks) takes 2-5 seconds. So <b>switching to an album by a different artist can take up to a minute</b>.</li>
</ul>
<li>Because the device needs to be reset constantly, loading up two different albums is a multi-minute ordeal that simply doesn't happen on the iPod or any smart phone.</li>
</ul>
<div>
It's a shame, because the audio quality is truly fantastic. But the device itself is unusable.</div>
<div>
<br /></div>
</div>
rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-46941159577896115082016-12-01T20:12:00.002-08:002016-12-01T20:32:03.470-08:00AMA Part 2: Software Development Resources<div class="tr_bq">
In <a href="http://www.roggr.com/2016/12/ama-part-1-coding-bootcamps.html" target="_blank">the previous post</a>, I discussed a few ways in which I have found coding bootcamps to be inadequate. In this post, I will present a list of resources I have found very useful as a software engineer. Many are free, and most others can be found in used bookstores (or Amazon) for moderate prices. I'd love to hear other people's favorite resources, so please add yours in the comments or on Twitter.</div>
<h3>
General Computer Science and Programming</h3>
Disclaimer: I didn't study CS formally, so this list is short and almost certainly out of date. Please send me your suggestions!<br />
<br />
<ul>
<li>Aho, Lam, Sethi - <a href="https://www.amazon.com/Compilers-Principles-Techniques-Tools-2nd/dp/0321486811" target="_blank">Compilers: Principles, Techniques and Tools</a></li>
<li>Gamma, Helm, Johnson, Vlissides - <a href="https://www.amazon.com/Design-Patterns-Elements-Reusable-Object-Oriented/dp/0201633612/" target="_blank">Design Patterns</a> (the "Gang of Four" book)</li>
<li>Knuth - <a href="https://www.amazon.com/Computer-Programming-Volumes-1-4A-Boxed/dp/0321751043" target="_blank">The Art of Computer Programming</a></li>
<li><a href="https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/" target="_blank">MIT Open Courseware, EE and CS</a></li>
</ul>
<h3>
Language Specific</h3>
<h4>
Official Tutorials / Documentation</h4>
<ul>
<li>The Java Tutorial - <a href="https://docs.oracle.com/javase/tutorial/">https://docs.oracle.com/javase/tutorial/</a></li>
<li>PHP Manual - <a href="http://php.net/manual/en/index.php">http://php.net/manual/en/index.php</a></li>
<li>Python doc - <a href="https://docs.python.org/3/">https://docs.python.org/3/</a></li>
<li>Ruby doc - <a href="http://ruby-doc.org/">http://ruby-doc.org/</a></li>
</ul>
<br />
Note: I <b>strongly do not recommend </b>a very popular Ruby introduction, <b>Why's (Poignant) Guide to Ruby. </b>A lot of people seem to love it. I found it <i>way </i>too cute, unclear, trying too hard, and just bad.<br />
<h4>
Books</h4>
<ul>
<li>Kernighan, Richie - <a href="https://www.amazon.com/Programming-Language-Brian-W-Kernighan/dp/0131103628/" target="_blank">The C Programming Language</a></li>
<li>Liberty, Halpern - <a href="https://www.amazon.com/C-Standard-Library-Scratch/dp/0789721287" target="_blank">The C++ Standard Library From Scratch</a></li>
<li>Lippman - <a href="https://www.amazon.com/Essential-C-Stanley-B-Lippman/dp/0201485184/" target="_blank">Essential C++</a></li>
</ul>
<div>
<blockquote class="tr_bq">
Note: even if you don't program in C or C++, Kernighan & Richie is a great introduction to computer programming at a low level. The Lippman and Liberty, Halpern books go very well together, and are project-oriented walkthroughs of essential features. All three of these books are very short, but they pack a punch.</blockquote>
</div>
<ul>
<li>Evans, Flanagan - <a href="https://www.amazon.com/Java-Nutshell-Benjamin-J-Evans/dp/1449370829/" target="_blank">Java In A Nutshell</a></li>
<li>Flanagan - <a href="https://www.amazon.com/Java-Examples-Nutshell-David-Flanagan/dp/0596006209" target="_blank">Java Examples In A Nutshell</a></li>
<li>Me - <a href="https://github.com/rogthefrog/programming-basics-with-java" target="_blank">Programming Basics with Java</a></li>
</ul>
<blockquote>
Note: get <b>both </b><i>Nutshell </i>books<b>, </b>and read them in parallel. Mine is a free work-in-progress, aimed at people who have never programmed but want/need to learn Java.</blockquote>
<ul>
<li>Crockford - <a href="http://shop.oreilly.com/product/9780596517748.do" target="_blank">JavaScript: The Good Parts</a></li>
<li>Martelli - <a href="http://shop.oreilly.com/product/9780596100469.do" target="_blank">Python in a Nutshell</a></li>
<li>Thomas, Fowler, Hunt - <a href="https://www.amazon.com/Programming-Ruby-Pragmatic-Programmers-Second/dp/0974514055" target="_blank">Programming Ruby</a></li>
<li>Flanagan, Matsumoto - <a href="https://www.amazon.com/Ruby-Programming-Language-Everything-Need/dp/0596516177/" target="_blank">The Ruby Programming Language</a></li>
</ul>
<blockquote class="tr_bq">
Note: anything by <b>Flanagan</b>, <b>Fowler</b>, <b>Beck</b>, <b>Crockford </b>or<b> </b><b>Martelli </b>is worth reading.</blockquote>
<h3>
After You've Coded For A While</h3>
<ul>
<li>Fowler, Beck - <a href="https://www.amazon.com/Refactoring-Improving-Design-Existing-Code/dp/0201485672" target="_blank">Refactoring</a></li>
<li>McConnell - <a href="https://www.amazon.com/Code-Complete-Practical-Handbook-Construction/dp/0735619670/" target="_blank">Code Complete</a></li>
<li>Stroustrup - <a href="https://www.amazon.com/C-Programming-Language-4th/dp/0321563840" target="_blank">The C++ Programming Language</a></li>
</ul>
<blockquote class="tr_bq">
Note: I listed Stroustrup here because it's a pretty dense, dry read (do <b>not </b>read it as your first programming book) that requires a good idea of how things work under the hood. It's also not a practical introduction to C++ <i>programming</i>; it's a guide to the C++ <i>language, </i>and from that perspective it's full of vital insights about choices made when C++ was designed, which in turn makes you think about <i>what computer languages can do, </i>and the <i>various ways they do it</i>.</blockquote>
<h3>
Specific Topics</h3>
<div>
<br />
<ul>
<li>Sankoff - <a href="https://www.amazon.com/Time-Warps-String-Edits-Macromolecules/dp/0201078090/" target="_blank">Time Warps, String Edits, and Macromolecules</a></li>
</ul>
</div>
<br />rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-40438728540677904272016-12-01T19:36:00.005-08:002017-01-07T12:24:53.015-08:00AMA Part 1: Coding BootcampsA recent <a href="https://twitter.com/search?q=%23devdiscuss&src=typd" target="_blank">#DevDiscuss thread on Twitter</a> focused on developers' various <a href="https://twitter.com/ThePracticalDev/status/803780702818934784" target="_blank">education paths</a> into the profession. The discussion was lively and a lot of good questions, answers and experiences were shared.<br />
<br />
After the discussion ended, a few people contacted me to ask more questions about the industry, and two themes emerged: 1) the various, popular <a href="https://www.switchup.org/research/best-coding-bootcamps" target="_blank">coding bootcamps</a> that have flourished in the past few years and 2) what resources (books, online tutorials, etc) I would recommend for a new software developer to become well-rounded (and employable).<br />
<br />
I'm going to address each question here in case it helps more than the folks who DMed me on Twitter. This post will be a reflection on bootcamps, and <a href="http://www.roggr.com/2016/12/ama-part-2-software-development.html" target="_blank">the next</a> will collect programming resources I've found useful.<br />
<h3>
On Coding Bootcamps</h3>
First, a few disclaimers:<br />
<br />
<ul>
<li>I have never been a student or instructor at any coding bootcamp</li>
<li>I hae nothing against short, intensive programs to learn a skill--I've taken a total of two computer-related classes, and both were short, intensive, and bootcamp-ish.</li>
<li>I realize different people have different learning styles; some do best alone, reading tutorials/books and writing tons of test code; others thrive with videos or podcasts; others still benefit from the focus and/or collegial learning you get in a classroom setting; etc.</li>
<li>I was a teacher for a few years, in college and adult ed, in standard quarter/semester-long as well as intensive summer programs; I've also spent a lot of time in the classroom as a student</li>
<li>I think it's fantastic that so many people (and especially underrepresented groups) are learning to become software engineers</li>
<li>I don't find a formal computer-science education is a particularly good predictor of talent or success in the software industry. Some of the best engineers I've worked with were humanities majors or high-school grads; some of the worst had MAs in CompSci from Stanford; there have also been great formally-trained engineers and awful self-taught engineers</li>
<li>My experience with bootcamps comes from interviewing about thirty applicants, offering jobs to two, and being friends with a couple</li>
<li>Some bootcamps may be fantastic. I don't know all of them, far from it</li>
</ul>
<div>
With that out of the way, here are some observations I've made about coding bootcamps.</div>
<div>
<br /></div>
<div>
I found two areas where bootcamps seem to be falling short: <b>tool/technology independence</b>, and low-level technical basics (<b>how stuff actually works</b>).</div>
<h3>
Technology Independence</h3>
<div>
Presumably because software engineering is a vast subject and you need to carefully limit the scope of an introductory course, I found bootcamps teach their students exactly <b>one </b>way to do things. with carefully selected tools, but not:</div>
<div>
<ul>
<li>other ways to do the same thing (with other tools, or with no big frameworks at all); </li>
<li>why those tools were chosen over others; and</li>
<li>what to do when you have to deal with a novel situation that doesn't exactly fit the standard paradigm.</li>
</ul>
<div>
The education seems limited to a very expensive Rails or Angular (or whatever framework) tutorial, carefully keeping students down the garden path of a basic application. There are a lot of tutorials available for free online; the majority </div>
<div>
<br /></div>
<div>
The graduates I talked to had never been exposed to any other way of doing things than the Rails/Angular/whatever way, even though 1) there are <b>many, equally valid ways </b>to approach application development and 2) the vast majority of industry jobs involve mixed, <b>heterogeneous </b>assemblies of tools, practices, and code from different eras/styles/people, and <b>finding a chunk of code that's exactly like the tutorial </b>so you can comfortably understand and modify it <b>is the exception, </b>not the rule.</div>
<div>
<br /></div>
<div>
Students were able to tell me how they would use ActiveRecord to interact with a database and display a list of things in a Rails view, but were stumped when I added common variations to the data stack (e.g. combining data from a SQL database with a document store like ElasticSearch). And when I gave them pieces of existing, real-life code to pick apart and modify to implement a new or different feature, most of them remained stuck and unable to figure out a way to make any progress.</div>
<div>
<br /></div>
<div>
I'm not blaming them for not knowing how to use something they weren't taught (all devs have to pick up new technology all the time); what I'm deploring is that <b>the bootcamps didn't give them the mental tools and technical knowledge to reason their way out of a predicament</b>. </div>
<div>
<br /></div>
Learning software engineering is a skill that will last you a lifetime. Knowing how to crank out an app with today's popular tools is a lot less valuable. Crucially, engineering skills like experimentation, figuring out how a piece of code works, exposure to multiple ways to do something so you're never stuck in one pattern you don't completely understand, those are arguably <b>the hardest skills to learn on your own</b>, and where a classroom setting, peers and a teacher to answer your questions, would be most beneficial. Learning how to use a framework, library or tool is the kind of stuff anyone can do with a little time and a browser, and a classroom setting isn't all that necessary. </div>
<div>
<h3>
How Stuff Works</h3>
<div>
Another area where the bootcamp graduates I spoke to were <b>entirely unprepared </b>is the <b>underlying low-level technology </b>that makes a networked app work (web app or internet-enabled mobile app). I'm not talking about arcana of TCP packet management or running a DNS server--the very basics of how software executes on a machine and how network/internet requests are made: how your browser finds example.com, contacts it, requests stuff, receives said stuff, and displays it. The kind of thing you absolutely have to understand when you're troubleshooting a problem in your live app, or when you're setting up a CDN, or when you're doing Ajax calls to a third-party domain, or dealing with HTTPS, or redirecting people from one page of your app to another.</div>
<div>
<br /></div>
<div>
Anyone can write an app that handles ideal circumstances; <b>what makes an engineer valuable is their ability to fix it </b>when it misbehaves. None of the bootcamp graduates was able to reason through the network path or anatomy of a basic web request. Very few knew how headers and cookies work. That stuff isn't complicated, you just need to see it once to understand it; and it's very important in a world of open, unsecured wi-fi access points and personalized apps and services, so you know why putting credentials in a cookie on a non-HTTPS site is a bad idea. </div>
<div>
<br /></div>
<div>
You don't need to be an expert; but <b>not knowing the basics will absolutely hold a person back</b>. Yes, those things can be learned on the job, but <i>getting </i>that job will be tricky if your education hasn't given you any information at all about the building blocks of your day-to-day work.</div>
<h3>
Silver Lining</h3>
<div>
Bootcamps are not all bad. I've heard and seen a lot of great feedback from people who genuinely got a lot out of them. Many bootcamps have industry partnerships or placement programs that help their graduates get hands-on experience in real software shops. The advantages of collegial learning are undeniable. Some people thrive in the pressure of intense, brief immersion into a topic. And you've got to start somewhere. </div>
<div>
<br /></div>
<div>
The other good news is that some of what I discussed above can be remedied easily; the information can be absorbed and understood in a couple of days of guided study. </div>
</div>
<div>
<h3>
Conclusion</h3>
</div>
<div>
Beyond the specifics I outlined above, what bothers me the most about the bootcamps I've been exposed to is that <b>they both overpromise and underdeliver. </b>Some (many? all?) claim to prepare future devs for the job market, but the ones I've been exposed to fall far short. And given how they seem to aggressively recruit from underrepresented populations (I've met a lot of non-male, non-white students from those bootcamps), it feels like <b>the students are being sold a bill of goods </b>and the promise of a fun, fulfilling and lucrative career, and are likely to be <b>surprised and bitterly disappointed </b>once they start interviewing for software engineering jobs.<br />
<br />
I'd be happy to recommend a bootcamp education if I knew of one that gave its students <b>more than a tutorial</b>, and <b>included a survey of the basic technology </b>underlying the kind of software its graduates are taught to write. If anyone reading this has a recommendation, I'd love to hear about it. Find me <a href="https://twitter.com/roger_b_m" target="_blank">on Twitter @roger_b_m</a> or comment here.</div>
<div>
<br /></div>
<div>
<h3>
Update 1/7/2017</h3>
<blockquote class="twitter-tweet" data-lang="en">
<div dir="ltr" lang="en">
Stories from coding bootcamps: many people have good experiences, but job placement stats are misleading or fake.<a href="https://t.co/8JiW8ChASy">https://t.co/8JiW8ChASy</a> <a href="https://t.co/YEfsDNPi8g">pic.twitter.com/YEfsDNPi8g</a></div>
— Dan Luu (@danluu) <a href="https://twitter.com/danluu/status/817779856951775232">January 7, 2017</a></blockquote>
<br />
<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script></div>
rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-41000931198890802992016-11-23T20:02:00.003-08:002016-11-23T23:10:16.343-08:00Docker lessons learned 1 year inA little under a year ago, I started doing devops work for a startup (the Company) with very specialized needs. As it operates in a highly regulated sector, the company's access to their infrastructure is extremely restricted, to prevent accidental or malicious disclosure of protected information. Their in-house web apps and off-the-shelf on-prem software are deployed on a compliant PaaS (I'll call them "the Host", even though they offer vastly more than just hosting), which is very similar to <a href="https://www.heroku.com/" target="_blank">Heroku</a> and uses <a href="https://www.docker.com/" target="_blank">Docker</a> exclusively for all applications deployed on their private EC2 cloud. I knew about Docker but had never used it, and it's been an interesting few months, so I thought I'd write up some observations in case they help someone.<br />
<h3>
Topsy Turvy</h3>
If you're coming to Docker from a traditional ops shop, it's important to keep in mind that many of your old habits and best practices either don't apply or are flipped upside down in a Docker environment. For example, you're probably going to use config management with Chef or Ansible a lot less, and convert your playbooks into Dockerfiles instead. Ansible/Chef/etc is based on the assumption that infrastructure has some level of <b>permanence</b>: you stand up a box, set it up with the right services and configuration, and it will probably be there and configured when you get around to deploying your app to it. By contrast, in the Docker world, things are much more <b>just-in-time</b>: you stand up and configure your container(s) <b>while</b> deploying your app. And when you update your app, you just toss the old containers and build new ones.<br />
<br />
Another practice that may feel unnatural is the foregrounding of (the main) processes. On a traditional web server, you'd typically run nginx, some kind of app server, and your actual app, all in the background. Docker, on the other hand, tends to use a one-service-one-container approach, and because a container dies when its main process does, you have to have something running in the <b>foreground </b>(not daemonized) for your container to stay up. Typically that'll be your main process (e.g. nginx), or you'll daemonize your main process and have an infinite <span style="font-family: "courier new" , "courier" , monospace;">tail -f /some/log</span> as your main process.<br />
<br />
As a corollary, while traditional server setups often have a bunch of backgrounded services all logging to files, a typical Dockerized service will only have one log you care about (the one for your main process), and because a container is usually an ephemeral being, its local file system is best treated as disposable. That means not logging to files, but to stdout instead. It's great for watching what's happening now, but not super convenient if you're used to hopping on a box and doing quick greps and counts or walking through past logs when troubleshooting something that happened an hour ago. To do that, you have to deploy a log management system as soon as your app goes live, not after you have enough traffic and servers that server-hopping, <span style="font-family: "courier new" , "courier" , monospace;">grep</span> and <span style="font-family: "courier new" , "courier" , monospace;">wc</span> has become impractical. So get your logstash container ready, because you need it now, not tomorrow.<br />
<br />
It's a decidedly different mindset that takes some getting used to.<br />
<div>
<br /></div>
I was already on board with the "everything is disposable" philosophy of modern high-availability systems, so conceptually it wasn't a huge leap, but if you're coming from a traditional shop with bare-metal (or even VM) deployments, it's definitely a mental switch.<br />
<h3>
Twelve Factor App Conventions</h3>
This one is more specific to the Host than to Docker in general, but it's part of an opinionated movement in modern software dev shops that includes Docker (and Rails, and Heroku), so I'll list it here. The <a href="https://12factor.net/" target="_blank">Twelve-Factor App manifesto</a> is a practical methodology for building modern apps delivered over the web. There's a lot of good stuff in there, like the emphasis on <a href="https://12factor.net/dependencies" target="_blank">explicit declarations</a> or the importance of a <a href="https://12factor.net/dev-prod-parity" target="_blank">dev/stage environment matching production</a> closely. But there's also questionable dogma that I find technically offensive. Specifically, factor 3 holds that <a href="https://12factor.net/config" target="_blank"><b>configuration must be stored in the environment</b></a> (as opposed to config files or delivered over some service).<br />
<br />
I believe this is <b>wrong. </b>The app is software that runs in user space; the environment is a safe, hands-off container for the app. The environment and the app live at different levels of resolution: all the app stuff is inward-looking, only for and about the app; while the environment is outward-looking, configured with and exposing the right data for its guests (the apps and services running in the environment). Storing app-level (userspace) data in the environment is like trusting the bartender in a public bar with your specific drink preferences, and asking her what you like to drink (yes, this is a bad simile).<br />
<br />
In addition, the concerns, scope, skills, budget, toolsets, and personalities of the folks involved in app work tend to be different from those of people doing the environment (ops) stuff. And while I'm ecstatic that devs and ops people appear to finally be merging into a "devops" hybrid, there's a host of practical reasons to divide up the work.<br />
<br />
In practical terms, storing configuration in the environment also has significant drawbacks given the tools of the trade: people like me use <span style="font-family: "courier new" , "courier" , monospace;">grep</span> dozens of times every day, and <span style="font-family: "courier new" , "courier" , monospace;">grep</span>ping through a machine's environment comprehensively (knowing that env variables may have been set as different Unix users) is error-prone and labor-intensive for no discernible benefit. Especially when your app is down and you're debugging things under pressure. It's also very easy to deploy what's supposed to be a self-contained "thing" (your twelve-factor app) and see it fail miserably, because someone forgot to set the environment variables (which highlights the self-contradictory, leaky nature of that config-in-the-environment precept: if your app depends on something external to it (the environment), it's not self-contained).<br />
<br />
Another driver for the config-in-the-environment idea is to make sure developers don't store sensitive information like credentials, passwords, etc. in code that winds up in source control (and thus on every dev's computer, and potentially accidentally left in code you helpfully decided to open-source on GitHub). That makes a ton of sense and I'm all for it. But for practical purposes, this still means every dev who wants to do work on their local machine needs a way to get those secrets onto their computer, and there aren't a lot of really <a href="https://www.hashicorp.com/blog/vault.html" target="_blank">easy-to-use</a>, <a href="https://aws.amazon.com/kms/" target="_blank">auditable</a>, <a href="https://www.conjur.net/products/secrets-management" target="_blank">secure</a> and practical methods to share secrets. In other words, storing configuration in the environment doesn't solve a (very real) problem: it just moves it somewhere else, without providing a practical solution.<br />
<br />
You may find this distinction specious, backwards, antiquated, or whatever. That's fine. The environment is the wrong place to store userspace/app-specific information. Don't do it.<br />
<br />
That was a long-winded preamble to what I really wanted to discuss, namely the fact that the Host embraces this philosophy, and in quite a few instances it's made me want to punch the wall. In particular, the Host makes you set environment variables using a command-line client that's kind of like running remote ssh commands, meaning that values you set need to be escaped, and they don't always get escaped or unescaped the way you expect when you query them. So if you set an environment variable value to its current value as queried by the command-line client, you'll double-escape the value (e.g. "<span style="font-family: "courier new" , "courier" , monospace;">lol+wat</span>" gets first set as "<span style="font-family: "courier new" , "courier" , monospace;">lol\+wat</span>"; looking it up returns "<span style="font-family: "courier new" , "courier" , monospace;">lol\+wat</span>" (escaped); resetting it turns it into "<span style="font-family: "courier new" , "courier" , monospace;">lol\\\+wat</span>"; i.e. a <b>set-get-set operation isn't idempotent</b>). All this is hard-to-debug, painfully annoying, and completely unnecessary if the model wasn't so stupid about using the environment for configuration.<br />
<h3>
Dev == Prod?</h3>
One of the twelve-factor tenets is that <b>dev/stage should mirror production closely</b>. This is a very laudable goal, as it minimizes the risk of unexpected bugs due to environment differences (aka "but it worked on my machine"). It's especially laudable as a lot of developers (at least in Silicon Valley) have embraced OSX/macOS as their OS of choice, even though nobody deploys web apps to that operating system in production, which means there's always a non-zero risk of stuff that works on dev failing on production because of some incompatibility somewhere. This also means every dev wastes huge amounts of time getting their consumer laptop to masquerade an industrial server, using ports and casks and bottles and build-from-source and other unholy devices, instead of just, you know, doing the tech work on the same operating system you're deploying on, because that would mean touching Linux and ewww that's gross.<br />
<br />
Originally, the Company had wrapped its production apps into Docker container using the Host's standard Dockerfiles and Procfiles, but devs were doing work on their bare-metal Macs, which meant finding, installing and configuring a whole bunch of things like Postgres, Redis, nginx, etc. That's annoying, overwhelming for new employees (since the documentation or Ansible playbooks you have to do that work are <i>always</i> behind and out of date about what actually happens on dev machines), and a pain to keep up to date. Individual dev machines drift apart from each other, "it works on my machine (but nor on yours)" becomes a frequent occurrence, and massive amounts of time (and money) are wasted debugging self-inflicted problems that really don't deserve to be debugged when it's so easy to do it right with a Linux VM and Ansible playbooks, but that would mean touching Linux and ewww that's gross.<br />
<br />
So I was asked to wrap the dev environment into Dockerfiles, and ideally we'd use the same Dockerfile as production, so that dev could truly mirror prod and we'd make all those pesky bugs go away. Good plan. Unfortunately, though, I didn't find that to be practical in the Company's situation: the devs use a lot of dev-only tools (unit test harnesses, linters, debuggers, tracers, profilers) that we really do not want to have available in production. In addition, starting the various apps and services is also done differently on dev and prod: debug options are turned on, logging levels are more verbose, etc. So we realized and accepted the fact that <b>we just can't use the same Dockerfile on dev and on prod. </b>Instead, I've been building a custom parent image that includes the intersection of all the services and dependencies used in the Company's various apps, and converting each app's Dockerfile to extend that new base image. This significantly reduces the differences and copy-pasta between Dockerfiles, and will give us faster deployments, as the base image's file system layers are shared and therefore more likely to be cached.<br />
<h3>
Runtime v. Build Time</h3>
Back to Docker-specific bits, this one was a doozy. When building the dev Dockerfiles, I had split the setup between system-level configuration (in the Dockerfile) and app-specific setup (e.g. <span style="font-family: "courier new" , "courier" , monospace;">pip install</span>s, node module installation, etc), which lived in a bootstrap script executed as the Dockerfile's CMD. It worked well, but it felt inelegant (two places to look for information about the container), so I was asked to move the bootstrap stuff into the Dockerfile.<br />
<br />
The devs' setup requirements are fairly standard: they have their Mac tools set up just right, so they want to be able to use them to edit code, while the code executes in a VM or a Docker container. This means sharing the source code folder between the Mac host and the Docker containers, using the well-supported <span style="font-family: "courier new" , "courier" , monospace;">VOLUME</span> or <span style="font-family: "courier new" , "courier" , monospace;">-v</span> functionality. Because node modules and pip packages are app-specific, they are listed in various bog-standard <span style="font-family: "courier new" , "courier" , monospace;">requirements.txt</span> and <span style="font-family: "courier new" , "courier" , monospace;">package.json</span> files in the code base (and hence in the Mac's file system). As the code base is in a shared folder mounted inside the Docker container, I figured it'd be easy to just put the <span style="font-family: "courier new" , "courier" , monospace;">pip install</span> stuff in the Dockerfile and point it at the mounted directories.<br />
<br />
But that failed, every time. A <span style="font-family: "courier new" , "courier" , monospace;">pip install -e /somepath/</span> that was meant to install a custom library in editable mode (so it's <span style="font-family: "courier new" , "courier" , monospace;">pip</span>-installed the same way as on prod, but devs can live-edit it) failed every time, missing its <span style="font-family: "courier new" , "courier" , monospace;">setup.py</span> file, which is RIGHT THERE IN THE MOUNTED FOLDER YOU STUPID F**KING POS. A <span style="font-family: "courier new" , "courier" , monospace;">pip install -r /path/requirements.txt</span> also failed, even though 1) it worked fine in the bootstrap script, which is also in the same folder/codebase 2) the volumes were specified and mounted correctly (I checked from inside the container).<br />
<br />
That's when I realized <b>the difference between build time and runtime </b>in Docker. The stuff in the <b>Dockerfile </b>is read and executed at <b>build time, </b>so your app has what it needs in the container at runtime. <b>During build time, your container isn't really running</b>--a bunch of temporary containers briefly run so various configuration steps can be executed, and they leave file system layers behind as Docker moves through the Dockerfile. The volumes you declare in your Dockerfile and/or docker-compose.yml file are <b>mounted </b>as you'd expect (you can ssh into your container and see the mount points); but they are only <b>bound </b>to the host's shared folders <b>at runtime</b>. This means that commands in your <b>Dockerfile</b> (which are used <b>at build time</b>) <b>cannot view or access files in your shared Mac folder</b>, because <b>those only become available at runtime</b>.<br />
<br />
Of course you could just <span style="font-family: "courier new" , "courier" , monospace;">ADD</span> or <span style="font-family: "courier new" , "courier" , monospace;">COPY </span>the files you need from the Mac folder into the mounted directory, and do your <span style="font-family: "courier new" , "courier" , monospace;">pip install</span> in the Dockerfile that way. It works, but it feels kinda dirty. Instead, what we'll do is identify which <span style="font-family: "courier new" , "courier" , monospace;">pip </span>libraries are used by most services, and bake those into our base image. That'll shave a few seconds off the app deployment time.<br />
<br />
Editorializing a bit, while I (finally) understand why things behaved the way they did, and it's completely consistent with the way Docker works, I feel it's a design flaw and should not be allowed by the Docker engine. It violates the <a href="https://en.wikipedia.org/wiki/Principle_of_least_astonishment" target="_blank">least-surprise principle</a> in a major way: it does only <i>part </i>of what you think it will do (create folders and mount points). I'd strongly favor some tooling in Docker itself that detects cases like these and issues a <span style="font-family: "courier new" , "courier" , monospace;">WARNING</span> (or barfs altogether if there was a <span style="font-family: "courier new" , "courier" , monospace;">strict</span> mode).<br />
<h3>
<span style="font-family: inherit;">Leaky Abstractions and Missing Features</span></h3>
Docker aims to be a tidy abstraction of a self-contained black box running on top of some machine (VM or bare-metal). It does a reasonable job using its union file system, but the abstraction is leaky: the underlying machine still peeks through, and can bite you in the butt.<br />
<br />
I was asked to Dockerize an on-prem application. It's a Java app which is launched with a fairly elaborate startup script that sets various command-line arguments passed to the JVM, like memory and paths. The startup script is generic and meant to just work on most systems, no matter how much RAM they have or where stuff is stored in the file system. In this case, the startup script sets the JVM to use some percentage of the host's RAM, leaving enough for the operating system to run. It does this sensibly, parsing <span style="font-family: "courier new" , "courier" , monospace;">/proc/meminfo </span><span style="font-family: inherit;">and injecting the calculated RAM into a </span><span style="font-family: "courier new" , "courier" , monospace;">-Xmx</span><span style="font-family: inherit;"> argument.</span><br />
<span style="font-family: inherit;"><br /></span>
But when Dockerized, the container simply refused to run: the Host had allocated some amount of RAM to it, and the app's launcher was requesting 16 times more, because the <span style="font-family: "courier new" , "courier" , monospace;">/proc/meminfo</span> file was... the host EC2 instance's! Of course, you could say "duh, that's a layered file system, of course that's what it does" and you'd be right. But the point is that a Docker container is not a fully encapsulated thing; it's common enough to query your environment's available RAM, and a clean, encapsulated container system <b>should always give an answer that's reflective of itself</b>, not breaking through to the underlying hardware.<br />
<h3>
Curious "Features"</h3>
Docker's network management is... peculiar. One of its more esoteric features is the order in which ports get <span style="font-family: "courier new" , "courier" , monospace;">EXPOSE</span>d. I was working on a Dockerfile that was extending a popular public image, and I could not make it visible to the outside world, even though my ports were explicitly <span style="font-family: "courier new" , "courier" , monospace;">EXPOSE</span>d and mapped. My parent image was <span style="font-family: "courier new" , "courier" , monospace;">EXPOS</span>ing port 443, and I wanted to expose a higher port (4343). For independent reasons, the Host's system only exposes the first port it finds, even if several are <span style="font-family: "courier new" , "courier" , monospace;">EXPOSE</span>d; and because there's <a href="https://github.com/docker/docker/issues/8709" target="_blank">no UNEXPOSE</a> functionality, it seemed I'd have to forget about extending the public base image and roll my own so I could control the port.<br />
<br />
But the Host's bottomless knowledge of Docker revealed that Docker exposes ports in <b><i>lexicographic order</i>. </b>Not numeric. That means <b>3000 comes before 443. </b>So I could still EXPOSE port a high port (3000) as long as <b style="font-style: italic;">lexicographically </b>it appeared before the base image's port 443, and the Host would pick that one for my app.<br />
<br />
I still have a bruise on my forehead from the violent D'OHs I gave that day.<br />
<br />
On a slightly higher level than this inside-baseball arcana, though, this "feature" also shows how leaky the Docker abstraction is: a child image is not only highly constrained by what the parent image does (you can't close/unexpose/override ports the parent <span style="font-family: "courier new" , "courier" , monospace;">expose</span>s), it (or its auhor) needs to have intimate knowledge of its parent's low-level details. Philosophically, that's somewhat contrary to the Docker ideal of every piece of software being self-contained. Coming at it from the software world, if I saw a piece of object-oriented code with a class hierarchy where a derived class had to know, be mindful of, or override a lot of the parent class's attributes, that'd be a <a href="https://blog.codinghorror.com/code-smells/" target="_blank">code smell</a> I'd want to get rid of pretty quickly.<br />
<h3>
Conclusion: Close, But Not Quite There</h3>
There is no question Docker is a very impressive and useful piece of software. Coupled with great, state-of-the-art tooling (such as the container tools available from AWS and other places), and some detailed understanding of Docker internals, it's a compelling method for deploying and scaling software quickly and securely.<br />
<br />
But in a resource-constrained environment (a small team, or a team with no dedicated ops resource with significant Docker experience), I doubt I'd deploy Docker on a large scale until some of its issues are resolved. Its innate compatibility with ephemeral resources like web app instances also makes it awkward to use with long-running services like databases (also known as <i>persistence</i> layers, so you know they tend to stick around). So you'll likely end up with a mixed infrastructure (Docker for certain things, traditional servers for others; Dockerfiles here, Ansible there; git push deploys here, yum updates there), or experience the <strike>ordeal</strike> joy of setting up a database in Docker.<br />
<br />
Adding to the above, the Docker ecosystem also has a history of shipping code and tools with significant bugs, <a href="https://thehftguy.wordpress.com/2016/11/01/docker-in-production-an-history-of-failure/" target="_blank">stability problems</a>, or <a href="https://docs.docker.com/engine/breaking_changes/" target="_blank">non-backward-incompatible changes</a>. <a href="https://docs.docker.com/engine/installation/mac/" target="_blank">Docker for Mac</a> shipped out of beta with show-stopping, CPU-melting bugs. The super common use case of running apps in Docker on dev using code in a shared folder on the host computer was only resolved properly a few months ago; prior to that, <span style="font-family: "courier new" , "courier" , monospace;">inotify</span> events when you modified a file in a shared, mounted folder on the host would not propagate into the container, and so apps that relied on detecting file changes for hot reloads (e.g. webpack in dev mode, or Flask) failed to detect the change and kept serving stale code. Before Docker for Mac came out, the "solution" was to <a href="https://github.com/brikis98/docker-osx-dev" target="_blank"><span style="font-family: "courier new" , "courier" , monospace;">rsync </span>your local folder</a> into its alter ego in the container so the container would "see" the inotify events and trigger hot reloads; an ingenious, effective, but brittle and philosophically bankrupt solution that gave me the fantods.<br />
<br />
Docker doesn't make the ops problem go away; it just moves it somewhere else. Someone (you) still has to deal with it. The promise is ambitious, and I feel it'll be closer to delivering on that promise in a year or two. We'll just have to deal with questionable tooling, impenetrable documentation, and doubtful stability for a while longer.rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-14282635218433995622016-11-11T18:37:00.001-08:002016-11-11T18:37:26.559-08:00Book in progress: Programming BasicsI started teaching a friend Java, and figured I might as well share the notes I wrote with anybody who may want them. Feedback appreciated (do read the README first to get a sense of the goals and intended audience, though):<br />
<br />
<a href="https://github.com/rogthefrog/programming-basics-with-java">https://github.com/rogthefrog/programming-basics-with-java</a>rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-4273700033822981812016-09-02T14:20:00.002-07:002016-09-02T14:29:05.505-07:00Python import basics in plain EnglishIn my own experience learning Python, and that of others on Python teams I've worked with, a common hurdle is <b>understanding how Python does imports</b>.<br />
<br />
The basics are actually very simple, but the <a href="https://docs.python.org/2/tutorial/modules.html" target="_blank">documentation</a> tends to be a little neckbeardy and dense, and hard to grok if you're new to the language. So I thought I'd list common, <b>simple practical examples of Python imports </b>in case they help someone.<br />
<br />
To import data and functions from somewhere else (another <span style="font-family: "courier new" , "courier" , monospace;">.py</span> file in your project, a standard library like <span style="font-family: "courier new" , "courier" , monospace;">os</span>, or a third-party library you may have installed with <span style="font-family: "courier new" , "courier" , monospace;">pip</span>), you have the following options:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">import <module><module></module></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">import </span><span style="font-family: "courier new" , "courier" , monospace;"><module> </span><span style="font-family: "courier new" , "courier" , monospace;">as </span><span style="font-family: "courier new" , "courier" , monospace;"><other_name></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">from </span><span style="font-family: "courier new" , "courier" , monospace;"><module> </span><span style="font-family: "courier new" , "courier" , monospace;">import a, b, c</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">from </span><span style="font-family: "courier new" , "courier" , monospace;"><module> </span><span style="font-family: "courier new" , "courier" , monospace;">import *</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: inherit;">Let's look at what these options mean.</span><br />
<h3>
<span style="font-family: "courier new" , "courier" , monospace;">import <module></module></span><span style="font-family: "courier new" , "courier" , monospace;"><module></span></h3>
<span style="font-family: "courier new" , "courier" , monospace;">import </span><span style="font-family: "courier new" , "courier" , monospace;"><module> </span><span style="font-family: inherit;">means the </span><span style="font-family: inherit;">program you're in can access everything that is defined in </span><span style="font-family: "courier new" , "courier" , monospace;"><module></span><span style="font-family: inherit;"> (variables, classes, functions, etc), and you have to prepend "</span><span style="font-family: "courier new" , "courier" , monospace;"><module>.</span><span style="font-family: inherit;">" to those things. For example, if </span><span style="font-family: "courier new" , "courier" , monospace;"><module></module></span> is the "<span style="font-family: "courier new" , "courier" , monospace;">os</span>" library (it comes with Python), which defines a function called <span style="font-family: "courier new" , "courier" , monospace;">getpid</span> and a variable named <span style="font-family: "courier new" , "courier" , monospace;">name</span>, your program can do this:<br />
<br />
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace;">>>> import os</span></span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> os.name</span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace;">'posix'</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace;">>>> os.getpid()</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace;">51678</span></span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
This works with your own libraries too. Say you created a Python file named <span style="font-family: "courier new" , "courier" , monospace;">network_functions.py</span> which contains a constant named <span style="font-family: "courier new" , "courier" , monospace;">BANDWIDTH</span> and a method named <span style="font-family: "courier new" , "courier" , monospace;">connect(url)</span>, you can do:</div>
<div class="p1">
<br /></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> import network_functions</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> network_functions.BANDWIDTH</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">1024</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> network_functions.connect('https://google.com')</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">Connecting...</span></div>
<div class="p1">
<br /></div>
<div class="p1">
If you don't want to have to type out the whole prefix (which can get unwieldy if your imports are nested (the modules are subdirectories), e.g. <span style="font-family: "courier new" , "courier" , monospace;">import lib.network.connection_functions</span>), you have the following options:</div>
<h3>
<span style="font-family: "courier new" , "courier" , monospace;">import </span><span style="font-family: "courier new" , "courier" , monospace;"><module> </span><span style="font-family: "courier new" , "courier" , monospace;">as </span><span style="font-family: "courier new" , "courier" , monospace;"><other_name></span></h3>
<div class="p1">
This lets you use <span style="font-family: "courier new" , "courier" , monospace;"><other_name> </span>instead of the module's full name. </div>
<div class="p1">
<br /></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> import lib.network.connection_functions <b>as netfunc</b></span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> <b>netfunc.</b>BANDWIDTH</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">1024</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> <b>netfunc.</b>connect('https://google.com')</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">Connecting...</span></div>
<h3>
<span style="font-family: "courier new" , "courier" , monospace;">from </span><span style="font-family: "courier new" , "courier" , monospace;"><module> </span><span style="font-family: "courier new" , "courier" , monospace;">import a, b, c</span></h3>
<div class="p1">
This lets you import only what you need from module into your current program's namespace. This means <b>everything you imported from the external module can be called by its bare name in your program:</b></div>
<div class="p1">
<br /></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> from lib.network.connection_functions import BANDWIDTH, connect</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> BANDWIDTH</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">1024</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> connect('https://google.com')</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">Connecting...</span></div>
<h3>
<span style="font-family: "courier new" , "courier" , monospace;">from </span><span style="font-family: "courier new" , "courier" , monospace;"><module></span><module style="font-family: "courier new", courier, monospace;"> import *</module></h3>
<div class="p1">
Import <b>everything</b> from the imported module into your current program's namespace, so you can call everything from the module by its bare name. <b>This is strongly not recommended, </b>and I'll explain why.</div>
<div class="p1">
<b><br /></b></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> from lib.network.connection_functions import *</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> BANDWIDTH</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">1024</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> connect('https://google.com')</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">Connecting...</span></div>
<div class="p1">
<br /></div>
<div class="p1">
<b>This is almost never a good idea</b>, because you don't always know or control what is defined in an external module, and there can be <b>name collisions, </b>e.g. functions or variables with the same name, so you may not be using the variable or function you expect! For example:</div>
<div class="p1">
<br /></div>
<div class="p1">
In file <span style="font-family: "courier new" , "courier" , monospace;">helpers.py</span>:</div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">def connect(url):</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;"> print "Connecting to", url</span></div>
<div class="p1">
<br /></div>
<div class="p1">
In file <span style="font-family: "courier new" , "courier" , monospace;">network.py</span>:</div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">def connect(url):</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;"> print "Hacking into", url</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> from helpers import *</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> from network import *</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> connect('https://google.com')</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;"># which one is called?</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> from network import *</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> from helpers import *</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> connect('https://google.com')</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;"># which one is called?</span></div>
<div>
<br /></div>
<div class="p1">
This example may seem contrived, but it's very common to import a bunch of modules written by different people, and some variable or function names are common or obvious enough that they may appear several times in different modules. Why wouldn't they, after all? Joe doesn't know about Jill's (or your own) module, so they have no reason to coordinate and ensure they're not using the same function names. </div>
<div class="p1">
<b><br /></b></div>
<div class="p1">
<b>If you use <span style="font-family: "courier new" , "courier" , monospace;">from </span><span style="font-family: "courier new" , "courier" , monospace;"><module></span><span style="font-family: "courier new" , "courier" , monospace;"><module> import *</module></span> with several modules, the odds are very good you'll call a function and actually invoke one that's not the one you expect.</b> And that can be really tricky to debug. </div>
<div class="p1">
<br /></div>
<div class="p1">
So what should you do if you do need a bunch of functionality from a module and don't want to import every single function and variable by name with:</div>
<div class="p1">
<br /></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">from </span><span style="font-family: "courier new" , "courier" , monospace;"><module> </span><span style="font-family: "courier new" , "courier" , monospace;">import var1, var2, var3, fun1, fun2, fun4 # etc </span></div>
<div class="p1">
<br /></div>
<div class="p1">
It's simple! <b><span style="font-family: inherit;">Don't use </span><span style="font-family: "courier new" , "courier" , monospace;">from </span></b><span style="font-family: "courier new" , "courier" , monospace;"><b><module> </b></span><b><span style="font-family: "courier new" , "courier" , monospace;"><module>import *.</module></span></b> Instead, use <span style="font-family: "courier new" , "courier" , monospace;"><b>import </b></span><span style="font-family: "courier new" , "courier" , monospace;"><b><module> </b></span>and presto, your program can use everything from <span style="font-family: "courier new" , "courier" , monospace;"><module></span>, as long as you <b>prefix <span style="font-family: "courier new" , "courier" , monospace;"><module></span><span style="font-family: "courier new" , "courier" , monospace;"><module>.</module></span></b> before the names.</div>
<div class="p1">
<br /></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> import os</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> os.getpid()</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">'posix'</span></div>
<h3>
Handy Tips</h3>
<div class="p1">
Do you ever want to know the variables or functions defined in a module you imported without having to Google them? Just use <span style="font-family: "courier new" , "courier" , monospace;">vars</span> or <span style="font-family: "courier new" , "courier" , monospace;">dir</span>:</div>
<div class="p1">
<br /></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace;">>>> import os</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace;">>>> dir(os)</span></span></div>
<div class="p1">
</div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">['EX_CANTCREAT', 'EX_CONFIG', 'EX_DATAERR', 'EX_IOERR', 'EX_NOHOST', 'EX_NOINPUT', 'EX_NOPERM', 'EX_NOUSER', 'EX_OK', 'EX_OSERR', 'EX_OSFILE', 'EX_PROTOCOL', 'EX_SOFTWARE', 'EX_TEMPFAIL', 'EX_UNAVAILABLE', 'EX_USAGE', 'F_OK', 'NGROUPS_MAX', 'O_APPEND', 'O_ASYNC', 'O_CREAT', 'O_DIRECTORY', 'O_DSYNC', 'O_EXCL', 'O_EXLOCK', 'O_NDELAY', 'O_NOCTTY', 'O_NOFOLLOW', 'O_NONBLOCK', 'O_RDONLY', 'O_RDWR', 'O_SHLOCK', 'O_SYNC', 'O_TRUNC', 'O_WRONLY', 'P_NOWAIT', 'P_NOWAITO', 'P_WAIT', 'R_OK', 'SEEK_CUR', 'SEEK_END', 'SEEK_SET', 'TMP_MAX', 'UserDict', 'WCONTINUED', 'WCOREDUMP', 'WEXITSTATUS', 'WIFCONTINUED', 'WIFEXITED', 'WIFSIGNALED', 'WIFSTOPPED', 'WNOHANG', 'WSTOPSIG', 'WTERMSIG', 'WUNTRACED', 'W_OK', 'X_OK', '_Environ', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '_copy_reg', '_execvpe', '_exists', '_exit', '_get_exports_list', '_make_stat_result', '_make_statvfs_result', '_pickle_stat_result', ] # and a bunch more</span></span></div>
<div class="p1">
<br /></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;">>>> vars(os)</span></div>
<div class="p1">
</div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">{'WTERMSIG': <built-in function="" wtermsig="">, 'lseek': <built-in function="" lseek="">, 'EX_IOERR': 74, 'EX_NOHOST': 68, 'seteuid': <built-in function="" seteuid="">, 'pathsep': ':', 'execle': <function 0x10e0aade8="" at="" execle="">, '_Environ': <class 0x10e0a7f58="" at="" os._environ="">, ] # and a bunch more</class></function></built-in></built-in></built-in></span></span></div>
<h4>
What's Next?</h4>
<div class="p1">
In a later post, I'll cover other tricky aspects of imports, namely how Python maps <span style="font-family: "courier new" , "courier" , monospace;">import <module></module></span> to Python code files in directories, and how to debug <span style="font-family: "courier new" , "courier" , monospace;">ImportError: No module named </span><span style="font-family: "courier new", courier, monospace;"><module> </span>problems that can occur depending on what directories your files are in.</div>
<div class="p1">
<br /></div>
<div class="p1">
Hopefully that was helpful!</div>
rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-19070245731765252702016-04-18T22:04:00.002-07:002016-04-18T22:10:26.073-07:00Django, static assets, versioning, and WhiteNoiseI had an interesting time troubleshooting an issue with Django, WhiteNoise and static asset versioning. This may be obvious to experienced Django users, but not to me; I've maintained Flask and Rails apps before, but Django is a new beast. I'll document it here in case it helps somebody.<br />
<br />
My goal was to set up asset versioning in a Django app to serve static files as <span style="font-family: "courier new" , "courier" , monospace;">filename.somehash.js</span> instead of <span style="font-family: "courier new" , "courier" , monospace;">filename.js</span> (same with other file types like css, png, etc). This is standard practice; most modern frameworks have that capability, and different ways to do it.<br />
<br />
I had started using <a href="http://whitenoise.evans.io/en/stable/" target="_blank">WhiteNoise</a> because the internets suggested it was a much, much easier task than other ways to do it. I was hoping to do asset versioning and deploy a Cloudfront CDN at the same time, and WhiteNoise is set up to do just that.<br />
<br />
Once everything was set up according to the documentation, I ran <span style="font-family: "courier new" , "courier" , monospace;">python manage.py collectstatic</span> and saw the versioned file names getting generated. Checking the files themselves confirmed that. But when I loaded the app in a browser, only the unversioned file names were being requested.<br />
<br />
After much head-scratching, I found this was because the app templates reference the static files with the standard <span style="font-family: "courier new" , "courier" , monospace;">{% load static %}</span> method. The problem went away when I changed that to <span style="font-family: "courier new" , "courier" , monospace;">{% load static from staticfiles %} </span><span style="font-family: inherit;">as suggested in <a href="https://github.com/evansd/whitenoise/issues/43" target="_blank">this closed issue on the subject</a>. Note that I didn't try the other option mentioned in that issue, </span><span style="font-family: "courier new" , "courier" , monospace;">{% load staticfiles %}</span><span style="font-family: inherit;">, but that should also work. </span><br />
<span style="font-family: inherit;"><br /></span>
Once the app restarted, beautiful unique file names were being requested and served. But I was occasionally getting 500 errors. I traced those back to instances where the app and WhiteNoise were being asked to serve files that no longer exist. Those references to deleted js, css, etc. files didn't actually harm the app's functionality, but when WhiteNoise is asked to serve them, it throws an exception and causes the app to 500.<br />
<br />
That's not ideal behavior--my take is that 50x errors in a production app should never happen and always be handled gracefully when they do, so a library that <i>causes</i> 500s by actively raising exceptions rather than logging, catching and handling them gracefully isn't ideal. But them's the breaks, and I might yet submit a PR to the owner if I find the time.<br />
<br />
In this particular app's case, this behavior was <i>especially </i>non-ideal because some of these files were referenced in commented-out JavaScript, and not actually requested; it looks like WhiteNoise and/or Django greedily consider <b>anything </b>that looks like a static file path to be actually requested, even if it's in code that doesn't execute.<br />
<br />
The solution is simple--find all those dangling references and exterminate them! Use those 500s to your advantage by exercising the app and tailing your error logs. It's easy to argue that's something you should do no matter what, so it wasn't hard to convince the code owners it was the right thing to do.<br />
<br />
<br />rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-20868671623787478592016-01-04T10:25:00.002-08:002016-01-04T10:28:41.006-08:00Installing wxPython in a virtualenv on Centos 6.7I'm looking at wxPython to write a GUI for an app I'm working on, and as it turns out using wxPython with virtual environments isn't completely obvious. Hopefully someone finds this helpful.<br />
<br />
My distribution is <b>CentOS 6.7</b> with a hand-built Python 2.7.6.<br />
<br />
<b>Step 1: Install wxPython</b><br />
<br />
This will install wxPython in your system's default Python library directory (not the one you want).<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">$ sudo yum install wxPython</span><br />
<span style="font-family: Courier New, Courier, monospace;">$ sudo find / -name wx*.py</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">/usr/lib64/python2.6/site-packages/wxversion.py</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">/usr/lib64/python2.6/site-packages/wx-2.8-gtk2-unicode/wxPython/lib/wxpTag.py</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">/usr/lib64/python2.6/site-packages/wx-2.8-gtk2-unicode/wxPython/lib/wxPlotCanvas.py</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">/usr/lib64/python2.6/site-packages/wx-2.8-gtk2-unicode/wx/tools/XRCed/plugins/wxlib.py</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">/usr/lib64/python2.6/site-packages/wx-2.8-gtk2-unicode/wx/tools/Editra/src/wxcompat.py</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">/usr/lib64/python2.6/site-packages/wx-2.8-gtk2-unicode/wx/lib/wxcairo.py</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">/usr/lib64/python2.6/site-packages/wx-2.8-gtk2-unicode/wx/lib/wxpTag.py</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">/usr/lib64/python2.6/site-packages/wx-2.8-gtk2-unicode/wx/lib/wxPlotCanvas.py</span><br />
<br />
<b>Step 2: Create and activate your virtualenv</b><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">$ cd</span><br />
<span style="font-family: Courier New, Courier, monospace;">$ virtualenv -p /usr/bin/python2.7 venv</span><br />
<span style="font-family: Courier New, Courier, monospace;">$ source venv/bin/activate</span><br />
<br />
Importing wx will fail:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">$ python</span><br />
<span style="font-family: Courier New, Courier, monospace;">Python 2.7.6 (default, Dec 2 2013, 21:17:42) </span><br />
<span style="font-family: Courier New, Courier, monospace;">[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2</span><br />
<span style="font-family: Courier New, Courier, monospace;">Type "help", "copyright", "credits" or "license" for more information.</span><br />
<span style="font-family: Courier New, Courier, monospace;">>>> import wx</span><br />
<span style="font-family: Courier New, Courier, monospace;">Traceback (most recent call last):</span><br />
<span style="font-family: Courier New, Courier, monospace;"> File "<stdin>", line 1, in <module></module></stdin></span><br />
<span style="font-family: Courier New, Courier, monospace;">ImportError: No module named wx</span><br />
<div>
<br /></div>
<b>Step 3: Symlink wxPython into your virtualenv</b><br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">$ cd ~/venv/lib/python2.7/site-packages</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">$ ln -s /usr/lib64/python2.6/site-packages/wx-2.8-gtk2-unicode/wx</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">$ ln -s /usr/lib64/python2.6/site-packages/wx-2.8-gtk2-unicode/wxPython</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">$ ln -s /usr/lib64/python2.6/site-packages/wxversion.py</span><br />
<br />
<b>Step 4: Start coding!</b><br />
<br />
<span style="font-family: Courier New, Courier, monospace;">$ source venv/bin/activate</span><br />
<span style="font-family: Courier New, Courier, monospace;">$ python</span><br />
<span style="font-family: Courier New, Courier, monospace;">Python 2.7.6 (default, Dec 2 2013, 21:17:42) </span><br />
<span style="font-family: Courier New, Courier, monospace;">[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2</span><br />
<span style="font-family: Courier New, Courier, monospace;">Type "help", "copyright", "credits" or "license" for more information.</span><br />
<span style="font-family: Courier New, Courier, monospace;">>>> import wx</span><br />
<span style="font-family: Courier New, Courier, monospace;">>>> app = wx.App()</span><br />
<span style="font-family: Courier New, Courier, monospace;">>>> frame = wx.Frame(None, -1, 'lol')</span><br />
<span style="font-family: Courier New, Courier, monospace;">>>> frame.Show()</span><br />
<span style="font-family: Courier New, Courier, monospace;">True</span><br />
<span style="font-family: Courier New, Courier, monospace;">>>> app.MainLoop()</span><br />
<br />
Note: I only just started playing with wxPython, so there may be other symlinks required to make it work. Let me know if so, and I'll update the post.<br />
<div>
<br /></div>
<div>
<br /></div>
rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-31672193382002000162015-03-25T13:41:00.002-07:002015-03-25T13:41:27.453-07:00Recruiter isn't even trying anymoreI got this InMain on LinkedIn today.<br />
<br />
<blockquote class="tr_bq">
March 25, 2015, 1:33 PM<br />Dear Roger,<br />Trust this finds you in good health!! I wanted to share a Back End Engineer role with you with one of my direct clients , its a long term contract and location is Mountain View, CA. Here is the short story:<br /><br />The candidate should be able to quickly adapt to different environment/framework as the project needs.<br />These are minimum required skills.<br />Open source backend stack<br />Web backend frameworks in Java OR C#, OR Python OR Ruby, etc.<br />Database design and maintenance using SQL and/or NoSQL<br />Let me know if you think that this would be a fit as per your skills and expertise, if no, maybe you can refer someone !!<br />Best,<br />Gargi </blockquote>
<div>
<ul>
<li>Nothing about the company (industry? size? age? product?)</li>
<li>The technology stack is essentially "whatever" (but open source! Including C#!)</li>
<li>The data stack is essentially "whatever"</li>
<li>The role is web AND database design AND dba</li>
</ul>
<div>
I LOL'ed heartily as I marked the InMail as spam.</div>
</div>
rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-57840823006918616122015-02-19T15:39:00.001-08:002015-02-19T16:58:43.757-08:00Comcast liesVerbatim transcript of a chat session with Naval. I was having internet connectivity problems. Naval says (s)he resolved them, and then did the standard upsell trick, in which (s)he lied. Then when I noticed the problems weren't resolved, (s)he lied again.<br />
<h2>
Summary</h2>
<h3>
Lie #1: " you will get additional service at a cheaper cost."</h3>
The rep correctly showed my current service is 50m down and basic cable TV for $82 / mo. He said Comcast had an offer where I could get "additional service at a cheaper cost." The "cheaper cost" is $101.99 for first 12 months and then $126.99 from 13-24 months.<br />
<h3>
Lie #2: "Every thing is totally fine"</h3>
<div>
At the exact time Naval said "Every thing is totally fine," Comcast was having well-documented problems:<br />
<br />
https://downdetector.com/status/comcast-xfinity/san-francisco<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAhoLBzaaTf5GYeZASR65oO5p-XKfOAvZCEtpbxcnGcnjDM5qTtEoJtPWwc_lbt3rWN1rkPK6r0prqbL1U0YifLZkB3q9bUtu5-jmnRK6vG56_g8roOuMqj9LNEW5OL1W0RjXmK32WTNsR/s1600/comcast_problems.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAhoLBzaaTf5GYeZASR65oO5p-XKfOAvZCEtpbxcnGcnjDM5qTtEoJtPWwc_lbt3rWN1rkPK6r0prqbL1U0YifLZkB3q9bUtu5-jmnRK6vG56_g8roOuMqj9LNEW5OL1W0RjXmK32WTNsR/s1600/comcast_problems.png" height="320" width="189" /></a></div>
<br />
<h3>
Lie #3 (probably): "You will also check this problem on our site also"</h3>
The "server upgradation" and service degradation is not listed.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVPnYjstC2AgAz-ISThVxnkNz_Wc86xWn4GMnisnYEI7qI6JkdaAVbf2ZhW8aLIwHjUNRDDVIFge5U6Tsm0up61h5nuCT9NqdpcNaJFpANGA-HvBHTp7LGfulHj9z8JYZFtp_SeCGm0Ykt/s1600/comcast_problems_2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVPnYjstC2AgAz-ISThVxnkNz_Wc86xWn4GMnisnYEI7qI6JkdaAVbf2ZhW8aLIwHjUNRDDVIFge5U6Tsm0up61h5nuCT9NqdpcNaJFpANGA-HvBHTp7LGfulHj9z8JYZFtp_SeCGm0Ykt/s1600/comcast_problems_2.png" height="242" width="320" /></a></div>
<br />
<h2>
Trancript</h2>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_analyst" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Naval: </span>Roger,I have also checked your account and found that we have a great offer for you where you will get <b>additional service at a cheaper cost.</b></div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_user" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Roger_: </span>what is that?</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_analyst" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Naval: </span>I can see that currently<b> you are paying around $82 .00</b> plus rental of devices and getting 50mbps speed</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_user" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Roger_: </span>that's correct, yes</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_analyst" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Naval: </span>And limited basic package for cable in which you are getting 10</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_analyst" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Naval: </span>channels</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_analyst" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Naval: </span>However, I have a good deal for you, In this package you will get 140+ Cable Channels and 105 mbps Internet Speed & Unlimited Nationwide Talk and Text.</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_analyst" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Naval: </span>Double Speed!!</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_analyst" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Naval: </span>14 times channels!!</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_analyst" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Naval: </span>Sounds Good?</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_user" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Roger_: </span>"additional service at a cheaper cost." = what is the cost?</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_analyst" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Naval: </span><b>This package will cost you only $101.99 for first 12 months</b>, after that it will cost you $126.99 from 13-24 months.</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_user" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Roger_: </span><b>how is that cheaper than $82?</b></div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_analyst" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Naval: </span>In this you will get double speed and 14 times channel and unlimited text and talk.</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_user" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<b><span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit;">Roger_: </span>you said additional services at a cheaper cost.</b></div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_user" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<b><span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit;">Roger_: </span>then you listed additional services at a higher cost.</b></div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_analyst" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Naval: </span>We are so confident in our products and services that we would want you to try them - risk free. If you’re not satisfied and wish to cancel the services for any reason, you can do so in the first 30 days and get your money back.</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_user" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Roger_: </span>what you did above is called bait and switch and it's illegal in the US</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_user" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Roger_: </span>this is one of the reasons people hate Comcast so much.</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_analyst" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Naval: </span>Okay. I appreciate your decision.</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_user" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Roger_: </span>you just lied to me.</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_analyst" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Naval: </span>Your satisfaction is my priority. Is there anything else I can assist you with? I am more than glad to help you out further.</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_user" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Roger_: </span><img alt="" class="emoticonimg" src="https://www.comcastsupport.com/ChatEntry/js/emoticon/emoticons/new/face-grin.png" /></div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_user" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Roger_: </span>I am satisfied with the tech support, but I am not satisfied with being lied to.</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_analyst" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Naval: </span>Thank you.</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_analyst" role="alert" style="font-family: Arial, sans-serif; font-size: 13.1999998092651px; margin-bottom: 8px; padding: 0px 0px 0px 15px; text-indent: -15px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Naval: </span>It was a pleasure assisting and chatting with you today! Have a great day and thank you for choosing Comcast. If you have any further questions, please do not hesitate to give us a call at 1-877-870-4310 or visit us at www.comcast.net for technical support. We appreciate your business!</div>
<div aria-live="polite" class="ccMessage cc_msgtype_msg cc_usertype_user" role="alert" style="margin-bottom: 8px; padding: 0px 0px 0px 15px;">
<div style="font-family: Arial, sans-serif; font-size: 13.1999998092651px;">
<span aria-hidden="true" aria-live="off" class="ccMessagFrom" style="font-size: inherit; font-weight: bold;">Roger_: </span>you're funny.</div>
<div style="font-family: Arial, sans-serif; font-size: 13.1999998092651px;">
<br /></div>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: I am glad to have assisted and get your concern fully resolved for today! </span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: You have opened a window for Comcast Support and I don't want to miss an opportunity to support you. Are we still connected?</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: Roger, Is there anything else I can assist you with?</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: my internet problems aren't resolved</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: still happenin</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: g</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: lots of services are hanging, etc.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: Let me check that out for you. Would you mind waiting for a couple of minutes while I do the research?</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: sure</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;">user Roger_ has left room</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;">user Roger has entered room</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: Welcome Back!!</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: Roger, As I can check from here <b>your account is totally fine and you are getting good services</b>.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: <b>Please run a speedtest </b>for us to ascertain the speed you are receiving at the moment and send me the result link once you're done. Here's the link : http://speedtest.comcast.net/</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: starting speed test</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: Okay.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: haha</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: "<b>starting in 5900 seconds</b>"</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: <b>that's an hour and a half</b></span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: Please allow me a minute .</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: normally I use speedtest.net but it's unreachable for me today</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: I'll try this http://www.speakeasy.net/speedtest/</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: I just ran the test</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: http://imgur.com/eMiqGft</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: Roger, <b>you are facing this issue because our system is upgrading</b>.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: 8.80M down, 0.68 up</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: I just did a tracert to amazonaws and it fails</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: <b>Every thing is totally fine</b>.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: Tracing route to amazonaws.com [207.171.166.22] [formatting garbled, thanks cmd.exe, lots of timeouts]</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: <b>how can you say everything is totally fine when I'm getting 8M down and traceroutes are not working</b></span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: <b>You will get the right speed once the upgradation procedure is finished</b>.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: you're saying my entire set of problems is because you're upgrading something?</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: It is due to our server upgradation.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: No worries.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: what kind of server? I'd like to know more about this.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: Sure!!</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: please don't tell me no worries. I work on the internet. That's how I make a living. I have deliverables. If my internet is not working, I cannot work. This is a big deal.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: Roger, <b>You will also check this problem on our site also</b>.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: <b>I checked before this chat and it said there were no issues</b>.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: where is the status shown?</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: I am doing advanced trouble shooting steps from here.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;">user Roger_ has left room</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;">user Roger has entered room</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: Please check now.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: Please tell me the result.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: aha</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: now we're talking</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: ... maybe</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: nope.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: no better.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: Roger, Please provide me your reliable phone number.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: xxx xxx xxxx</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Roger_</b>: 20.96 down, 0.44 up</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: I am raising ticket for you.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: Our senior technical team will call you on the number given by you.</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><b>Naval</b>: Please note down your ticket number</span></span><br />
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span>
<br />
<div>
<span style="font-family: Arial, sans-serif;"><span style="font-size: 13.1999998092651px;"><br /></span></span></div>
</div>
</div>
<h2>
Update</h2>
I spoke with another rep, Jose, who was great. Excerpts from our chat, mostly for the lulz:
<br />
<br />
<blockquote class="tr_bq">
Jose: May I know the issue that our rep lied to you?<br />
Roger: (summary of the above)<br />
Roger: I asked the price<br />
Roger: "This package will cost you only $101.99 for first 12 months, after that it will cost you $126.99 from 13-24 months."<br />
Roger: $101.99 is not actually cheaper than $91.96.<br />
Jose: yes that is correct. Its simple math.</blockquote>
<div>
<br /></div>
<div>
Then this unintentionally hilarious pearl:</div>
<div>
<div>
<br /></div>
<blockquote class="tr_bq">
Jose: On internet issue they can be corrected by our Internet dept.<br />
Jose: We are cable troubleshooting dept so all of our tools here are for cable TV.<br />
Jose: <b>On broken promises those usually happens on Sales dept.</b>Jose: Here on cable troubleshhoting we dont lie.</blockquote>
</div>
<div>
<br /></div>
<div>
<br /></div>
rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-8176899036279100492014-06-19T13:29:00.005-07:002014-06-19T13:30:35.710-07:00Enabling LZO compression for Hive to avoid cannot find class com.hadoop.mapred.DeprecatedLzoTextInputFormat errorI just spent a bunch of time reading through documentation and Google Group postings about how to enable LZO compression in Hive, only to find none of them was the right solution. In the end I did find something that worked, so hopefully this can help someone.<br />
<br />
<b>Goal</b>: enable LZO-compressed files to be used for Hive tables.<br />
<br />
<b>Environment</b>: Hadoop cluster managed with Cloudera Manager version 5.<br />
<br />
<b>Prerequisites</b>:<br />
<ul>
<li>install and activate the parcel that contains the LZO library <a href="http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Installation-Guide/cm5ig_install_gpl_extras.html" target="_blank">as shown here</a></li>
<li>configure it <a href="http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Managing-Clusters/cm5mc_gpl_extras.html" target="_blank">as shown here</a></li>
</ul>
What's missing from the instructions and <a href="https://groups.google.com/a/cloudera.org/forum/#!msg/cdh-user/5NMbUIeMX0c/dm7XyPKtkQMJ" target="_blank">the Google Group postings about that error</a> is <b>how to tell Hive where to find the Hadoop LZO jar</b>. The instructions about classpath settings above are not sufficient, and you'll have this error when running a Hive query against an LZO table:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">cannot find class com.hadoop.mapred.DeprecatedLzoTextInputFormat</span> <br />
<br />
To fix this:<br />
<ul>
<li>Go to your Cloudera Manager UI home page</li>
<li>Click <b>Hive</b></li>
<li>Click <b>Configuration </b>> <b>View and Edit</b></li>
<li>Under <b>Service-Wide</b> > <b>Advanced</b>, look for<b> Hive Auxiliary JARs Directory</b></li>
<li>Set the value to <span style="font-family: "Courier New",Courier,monospace;">/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib</span></li>
<li>Restart the Hive service (and any related services)</li>
</ul>
Now you can run queries against LZO-compressed files.<br />
<br />
As a reminder, to create a table backed by LZO-compressed files in HDFS, do something like this:<br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span><span style="font-family: "Courier New",Courier,monospace;">CREATE EXTERNAL TABLE `my_lzo_table`(`something` string)<br />ROW FORMAT DELIMITED <br /> FIELDS TERMINATED BY '\t' <br />STORED AS INPUTFORMAT <br /> 'com.hadoop.mapred.DeprecatedLzoTextInputFormat' <br />OUTPUTFORMAT <br /> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'<br />LOCATION<br /> '/hdfs/path/to/your/lzo/files';</span>rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-18101822040296680742014-06-11T18:04:00.001-07:002014-06-11T18:04:42.650-07:00Cloudera Manager Fails to RestartI've been experimenting with <a href="http://www.cloudera.com/content/cloudera/en/products-and-services/cloudera-enterprise/cloudera-manager.html" target="_blank">Cloudera Manager</a> to manage Hadoop clusters on EC2. So far it seems to be working a little better than <a href="http://ambari.apache.org/" target="_blank">Ambari</a>, which managed to install its agent software on all my nodes but always failed to start the required services.<br />
Cloudera Manager did fail as well, but that seemed to be due to my security group settings. I changed the configuration and restarted the service on the host, but the restart always failed with this error:<br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">Caused by: java.io.FileNotFoundException: /usr/share/cmf/python/Lib/site$py.class (Permission denied)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> at java.io.FileInputStream.open(Native Method)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> at java.io.FileInputStream.<init>(FileInputStream.java:146)</init></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> at org.hibernate.ejb.packaging.ExplodedJarVisitor.getClassNamesInTree(ExplodedJarVisitor.java:126)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> at org.hibernate.ejb.packaging.ExplodedJarVisitor.getClassNamesInTree(ExplodedJarVisitor.java:134)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> at org.hibernate.ejb.packaging.ExplodedJarVisitor.getClassNamesInTree(ExplodedJarVisitor.java:134)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> at org.hibernate.ejb.packaging.ExplodedJarVisitor.doProcessElements(ExplodedJarVisitor.java:92)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> at org.hibernate.ejb.packaging.AbstractJarVisitor.getMatchingEntries(AbstractJarVisitor.java:149)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> at org.hibernate.ejb.packaging.NativeScanner.getClassesInJar(NativeScanner.java:128)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> ... 31 more</span><br />
<br />
Odd, I thought, since by default the service runs as root and should have free rein. So I poked around in that Python library directory, and lo and behold:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEOBdSvMMY64RuI-Ik7LutxJFHnsWC5d1l66CoW36cmjIgGrlKEUKAc_EHK6V_cXNbj_255mq050tSi5eqp8x4et8ApEzsvykXaYHqiFGF7O5UaEawlZDi3jlRVnnKg9r667-Imv0B7F7e/s1600/20140611_cloudera_wrong_permissions.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEOBdSvMMY64RuI-Ik7LutxJFHnsWC5d1l66CoW36cmjIgGrlKEUKAc_EHK6V_cXNbj_255mq050tSi5eqp8x4et8ApEzsvykXaYHqiFGF7O5UaEawlZDi3jlRVnnKg9r667-Imv0B7F7e/s1600/20140611_cloudera_wrong_permissions.png" height="285" width="320" /></a></div>
I chmod'ed 644 the .class files (in /usr/share/cmf/python/Lib and /usr/share/cmf/python/Lib/simplejson) and sure enough everything is working again.<br />
Hopefully this is helpful to somebody.rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com5tag:blogger.com,1999:blog-7042147755594594926.post-20827715981898411692014-05-26T15:30:00.002-07:002014-05-28T13:57:29.221-07:00Installing Scrapy on an Amazon CentOS AMI<a href="http://parelabs.com/" target="_blank">We</a>'re experimenting with Scrapy, and I thought I'd share what I found while installing the Scrapy package, as it has multiple dependencies many Python installations don't normally include, and those are <a href="http://doc.scrapy.org/en/latest/intro/install.html#intro-install" target="_blank">not listed in the documentation</a>.<br />
<br />
* First off, you want bzip2:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ cd /tmp</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ wget http://bzip.org/1.0.6/bzip2-1.0.6.tar.gz<br />$ tar -xzf bzip2-1.0.6.tar.gz<br />$ cd bzip2-1.0.6<br />$ sudo make -f Makefile-libbz2_so<br />$ sudo make<br />$ sudo make install PREFIX=/usr/local<br />$ sudo cp libbz2.so.1.0.6 /usr/local/lib</span><br />
<br />
<br />
* Then you want the libffi headers<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ sudo yum install libffi-devel</span><br />
<br />
* Then you want to download the Python source and extract it:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ wget https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tgz</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ tar -xzf Python-2.7.6.tgz</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ cd Python-2.7.6</span><br />
<br />
* I usually uncomment any lines referencing ssl and zlib in Modules/Setup.dist <br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ vi Modules/Setup.dist</span><br />
<span style="font-family: "Courier New",Courier,monospace;"># find and uncomment the lines</span><br />
<br />
* Then build Python:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ ./configure --prefix=/usr/local</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ sudo make</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ sudo make altinstall</span><br />
<br />
* <a href="http://virtualenv.readthedocs.org/en/latest/virtualenv.html#installation" target="_blank">Install virtualenv</a> if you don't have it (you should)<br />
<br />
* Activate your virtualenv with your fresh Python<br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;">$ cd</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ virtualenv -p /usr/local/bin/python2.7 myenv</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ source myenv/bin/activate</span><br />
<br />
* Install Scrapy<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ easy_install Scrapy</span><br />
<br />
This was tested on an Amazon AWS image named amzn-ami-pv-2013.09.2.x86_64-ebs (ami-a43909e1), known as Amazon Linux AMI x86_64 PV EBS with the following version strings:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ cat /proc/version <br />Linux version 3.4.73-64.112.amzn1.x86_64 (mockbuild@gobi-build-31003) (gcc version 4.6.3 20120306 (Red Hat 4.6.3-2) (GCC) ) #1 SMP Tue Dec 10 01:50:05 UTC 2013</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ cat /etc/*-release<br />Amazon Linux AMI release 2014.03</span>rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-52730717767329076122014-04-24T17:10:00.002-07:002014-04-24T17:10:40.594-07:00Elastic MapReduce, Hive and Input Files<a href="http://parelabs.com/" target="_blank">We</a>'re using <a href="http://hive.apache.org/" target="_blank">Hive</a> and Amazon's <a href="https://aws.amazon.com/elasticmapreduce/" target="_blank">Elastic MapReduce</a> to process sizable data sets. Today, I was wondering why a simple count query on a table with under a billion rows was taking a long time. The table file is in a single gzipped file in an S3 bucket, and Hive was only using a single mapper. So I thought, hrm, it looks like the job isn't distributed at all, so let's try splitting the input file into a bunch of smaller files to see if Hive will be able to put more mappers to work.<br />
<br />
This is the initial slow job, with a single gzipped file for the table in S3:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">-- SINGLE .gz FILE AS HIVE TABLE</span><br />
<span style="font-family: Courier New, Courier, monospace;">hive> select count(*) FROM mytable;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">Job 0: Map: 1 Reduce: 1 Cumulative CPU: 254.84 sec HDFS Read: 207 HDFS Write: 10 SUCCESS</span><br />
<span style="font-family: Courier New, Courier, monospace;">Total MapReduce CPU Time Spent: 4 minutes 14 seconds 840 msec</span><br />
<span style="font-family: Courier New, Courier, monospace;">OK</span><br />
<span style="font-family: Courier New, Courier, monospace;">239370915</span><br />
<span style="font-family: Courier New, Courier, monospace;">Time taken: 274.51 seconds, Fetched: 1 row(s)</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: inherit;">This is the same job run against 240 non-gzipped files for the table in S3:</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">-- MULTIPLE FILES, not gzipped</span><br />
<span style="font-family: Courier New, Courier, monospace;">hive> select count(*) FROM mytable_multiple_files_no_gzip;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">Job 0: Map: 48 Reduce: 1 Cumulative CPU: 538.05 sec HDFS Read: 25536 HDFS Write: 10 SUCCESS</span><br />
<span style="font-family: Courier New, Courier, monospace;">Total MapReduce CPU Time Spent: 8 minutes 58 seconds 50 msec</span><br />
<span style="font-family: Courier New, Courier, monospace;">OK</span><br />
<span style="font-family: Courier New, Courier, monospace;">239370915</span><br />
<span style="font-family: Courier New, Courier, monospace;">Time taken: 55.071 seconds, Fetched: 1 row(s)</span><br />
<br />
Not bad, eh?<br />
<br />
Then I tried the same split schema, except each file was gzipped individually (240 gzipped input files):<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">-- MULTIPLE FILES, gzip</span><br />
<span style="font-family: Courier New, Courier, monospace;">hive> select count(*) FROM mytable_multiple_files_gzip;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: 'Courier New', Courier, monospace;">Job 0: Map: 240 Reduce: 1 Cumulative CPU: 1552.43 sec HDFS Read: 52080 HDFS Write: 10 SUCCESS</span><br />
<span style="font-family: Courier New, Courier, monospace;">Total MapReduce CPU Time Spent: 25 minutes 52 seconds 430 msec</span><br />
<span style="font-family: Courier New, Courier, monospace;">OK</span><br />
<span style="font-family: Courier New, Courier, monospace;">239370915</span><br />
<span style="font-family: Courier New, Courier, monospace;">Time taken: 112.735 seconds, Fetched: 1 row(s)</span><br />
<br />
So with gzipped input files, I had a one mapper-one file relationship; with uncompressed input files, I had a one mapper-five files relationship.<br />
<br />
These numbers were obtained on a cluster with 8 i2.2xlarge data nodes and an m3.xlarge name node.<br />
<br />
Typically (at least that's what a cursory Google search suggests), people have <a href="http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-hive-differences.html" target="_blank">the opposite problem</a>--too many small-ish files in S3, and too many mappers. Too many mappers can delay your reducers' work. So I'll do some testing on different splitting schemas for the same data set and update.<br />
<br />rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-7532266277841036712014-04-04T15:38:00.000-07:002014-04-04T15:57:40.980-07:00McCarthy was self-righteous tooBrendan Eich, inventor of JavaScript, <a href="http://online.wsj.com/news/articles/SB10001424052702303532704579479741125367618" target="_blank">just resigned from his brand new position as CEO of the Mozilla foundation</a>, after it was discovered he made a $1000 donation to the anti-gay-marriage campaign in California known as Prop 8.<br />
<br />
That discovery caused uproar among the self-righteous <i>bien-pensants</i> who work for Mozilla, and a number of employees posted tweets about how they thought he should resign.<br />
<br />
I'm angry about this because this isn't very different from McCarthyism in reverse. A guy was forced out of a job because his political views don't agree with the majority.<br />
<br />
I feel opposing gay marriage is bigoted, wrong, indefensible and on the wrong side of history. I don't know Eich. For all I know he's a raging asshole with ultra-right-wing views. He might even hate kittens and burp at the dinner table. I don't know.<br />
<br />
But what I do know is that getting forced out of a job by a self-righteous San Francisco mob of entitled nerds who have probably never even seen a Republican in the flesh is just as indefensible. It's not what America and California are about. And it shows liberals can be assholes, too, when they put their minds to it.<br />
<br />
I'd venture to say a very large number of CEOs are raging right-wing Republicans with questionable ethics. If you don't like your CEO's politics, you're free to work somewhere else. Your job isn't in grave danger if you and your CEO don't see eye-to-eye in terms of politics--there are laws on the books protecting you from discrimination. Why should your CEO's job be in jeopardy for that very same reason?<br />
<br />
Eich's contributions to Web tech are immense and he may well be as capable as anyone of running Mozilla, a company he's been with for years. Yet he lost his job because of his politics. And that's not right, whether you agree with him or not.rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com1tag:blogger.com,1999:blog-7042147755594594926.post-78850955853144712062014-03-12T13:13:00.001-07:002014-03-13T17:29:08.955-07:00Notes on AzkabanI've been evaluating tools to run data processing jobs and narrowed my list down to <a href="https://github.com/spotify/luigi" target="_blank">Luigi</a> and <a href="https://github.com/azkaban/azkaban" target="_blank">Azkaban</a>.<br />
<br />
I nixed Luigi for a number of reasons:<br />
<ul>
<li>you can't execute jobs from the web UI.</li>
<li>you can't schedule jobs--you still have to use cron and all the bs that goes with that (manually managing overlap, e.g. what should I do when my first job is still running when the second job is scheduled to start?).</li>
<li>the <a href="https://github.com/spotify/luigi/blob/master/README.md" target="_blank">documentation</a> is horrid.</li>
</ul>
<div>
So far so good. Azkaban does have some quirks I'm working through. For example:</div>
<div>
<ul>
<li>the executor.host property is not in the default config. The web component wisely defaults to localhost, but it would be handy to have it in the default config, even commented out (like many other properties) so you can run Azkaban in its preferred distributed mode without having to look through Google Groups questions.</li>
<li><strike>I still can't figure out how to set up the host configuration for the Hadoop cluster Azkaban is supposed to talk to</strike>. Fixed--see below.</li>
</ul>
<div>
But the UI is intuitive and it handles the overlap issue (for a given job flow) like a champ: if a job is schedule or run while it's still running, you can tell Azkaban to abort the second run, let it run in parallel, or wait until the first run completes before the second run starts.</div>
</div>
<div>
<br />
Things to remember:<br />
<ul>
<li><b>Hadoop and Hive must be installed on the job executor box</b>. Not running, just installed, with the standard <span style="font-family: Courier New, Courier, monospace;">HADOOP_HOME</span> and <span style="font-family: Courier New, Courier, monospace;">HIVE_HOME</span> env vars set, etc. </li>
<li>Then you have to <b>put your actual cluster's config files in the executor server's Hadoop config directory</b> (typically <span style="font-family: Courier New, Courier, monospace;">$HADOOP_HOME/conf</span>). This is because Azkaban looks for your <b>remote</b> Hadoop name node location in its <b>local</b> Hadoop configuration files. </li>
<li>There's a lot of documentation out there based on Hadoop's old (1.x) directory structure. Hadoop 2.x has changed a lot of that and the Jars aren't where you'd expect them. <b>Inspect your classpaths</b> in all the config and properties files used by Azkaban and your Hive jobs. If a Hive job fails, it's a good bet you have a classpath problem, so <b>look at your Azkaban executor server logs </b>(not just the logs in the web interface).</li>
<li>The <b>startup and shutdown scripts in bin/ are pretty brittle</b>. Make sure all the directories are set correctly and you handle errors if they're not. Also, make sure to <b>run them as <span style="font-family: Courier New, Courier, monospace;">bin/script.sh</span></b> instead of <span style="font-family: Courier New, Courier, monospace;">cd bin; ./script.sh</span> because they rely on relative directories.</li>
<li>Remember to <b>open port 12321 </b>in the executor server's firewall so the web server can submit jobs.</li>
<li>Remember to open <b>port 9000 </b>on your master Hadoop node so Azkaban can submit jobs to it. </li>
<li><b>One project == one flow</b>. If you upload more than one flow into a project, only the last one is retained. </li>
<li><b>You can't run <a href="https://github.com/azkaban/azkaban-plugins/tree/master/plugins/jobtype/examples/" target="_blank">the Hive job examples</a> as-is</b>. They won't work because they're missing a few properties.</li>
</ul>
<div>
This is the basic <a href="https://github.com/azkaban/azkaban-plugins/tree/master/plugins/jobtype/examples/hive-wc" target="_blank">Hive word count job</a> example:</div>
<div>
<br /></div>
<div>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">type=hive<br />user.to.proxy=azkaban<br />hive.script=scripts/hive-wc.hql</span></blockquote>
</div>
<div>
<br /></div>
<div>
In order for it to work, it needs to look like this (differences that matter are bolded):</div>
<div>
<br /></div>
<div>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">type=hive<br />user.to.proxy=hadoop<br /><b>azk.hive.action=execute.query</b><br /><b>classpath=./*,./lib/*,${hadoop.home}/*,${hadoop.home}/lib/*,${hive.home}/lib/*<br />hive.query.file</b>=scripts/hive-wc.hql</span></blockquote>
<br />
I'm using Azkaban 2.1 and Hadoop 1.2.1. Different versions will have different paths and classpaths, but the <span style="font-family: Courier New, Courier, monospace;">azk.hive.action</span>, <span style="font-family: Courier New, Courier, monospace;">classpath</span> and <span style="font-family: Courier New, Courier, monospace;">hive.query.file</span> are crucial (<span style="font-family: Courier New, Courier, monospace;">hive.script</span> doesn't work).<br />
<br />
Other things that don't work out of the box:<br />
<br />
<ul>
<li>When proxying to use the hadoop user of your choice, you need to set the security manager class to its fully qualified name. The sample config does not fully qualify the class name and so the executor fails to load. To wit:</li>
</ul>
<div>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"># hadoop security manager setting common to hadoop jobs<br />hadoop.security.manager.class=HadoopSecurityManager_H_1_0</span></blockquote>
</div>
<div>
<br /></div>
<div>
should be</div>
<div>
<br /></div>
<div>
<blockquote class="tr_bq">
<span style="font-size: x-small;"><span style="font-family: Courier New, Courier, monospace;"># hadoop security manager setting common to hadoop jobs </span><span style="font-family: Courier New, Courier, monospace;">hadoop.security.manager.class=<b><span style="color: red;">azkaban.security.</span></b>HadoopSecurityManager_H_1_0</span></span></blockquote>
</div>
<div>
<br /></div>
<div>
in the <span style="font-family: Courier New, Courier, monospace;">plugins/jobtypes/commonprivate.properties</span> file.</div>
<div>
<br /></div>
</div>
</div>
rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com1tag:blogger.com,1999:blog-7042147755594594926.post-23580848094558303402013-08-15T17:58:00.001-07:002013-08-15T17:58:32.870-07:00OCDThe deepest cruelty of OCD is that the D stands for disorder.rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-49562757377503903902013-07-10T18:48:00.003-07:002013-07-10T18:54:42.252-07:00A painless git workflowI was thrown into the Rails and git worlds a couple of years ago when joining Crunched, and while Ruby was a fairly painless experience, git was heinous for about a year. Then it clicked, mostly thanks to <a href="https://twitter.com/plukevdh" target="_blank">Luke</a> and <a href="https://twitter.com/tjsingleton" target="_blank">TJ</a> and a good workflow. So I figured I'd share what we settled on. Maybe it'll same someone some grief.<br />
<br />
<br />
<iframe src="http://www.slideshare.net/slideshow/embed_code/24121893" width="476" height="400" frameborder="0" marginwidth="0" marginheight="0" scrolling="no"></iframe>
<br />
<br />
Static PNGs below:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhA6paBhrVrOluKi1nLtxhPUMJUkE7-XfqMMv2xW_kUfWnpwP4bhZnSiebKNsyTKS9NTSvlhyphenhyphenWU_uHMYSBIQnbfXafVokjENONZ5ft2P05BAtluRP2uffyx99itZT8w5nkddlVQ9lwkmCz0/s1600/20130710_git_workflow_0.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="199" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhA6paBhrVrOluKi1nLtxhPUMJUkE7-XfqMMv2xW_kUfWnpwP4bhZnSiebKNsyTKS9NTSvlhyphenhyphenWU_uHMYSBIQnbfXafVokjENONZ5ft2P05BAtluRP2uffyx99itZT8w5nkddlVQ9lwkmCz0/s320/20130710_git_workflow_0.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitfnPlVa_uePi5HBP-YqROYWsJZFpVMbvzku8ajEvPn9oSnjOW4pnF9yPbo_JurkGWub8XLfovmsrPAVUfVEIuqTOyMH5TF516KJQkEC2zBz_Oye0H4hbVrLGdzirlHZdOAopy92oEG2O5/s1600/20130710_git_workflow_1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="182" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitfnPlVa_uePi5HBP-YqROYWsJZFpVMbvzku8ajEvPn9oSnjOW4pnF9yPbo_JurkGWub8XLfovmsrPAVUfVEIuqTOyMH5TF516KJQkEC2zBz_Oye0H4hbVrLGdzirlHZdOAopy92oEG2O5/s320/20130710_git_workflow_1.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRzYzJB6xWMDDfi-m3nfGPJtF_BVh-wrCsjMzmHeQanGw7-JkLq7l9_UntYPGyV1ywYkc0iAyCZYMTufCyrVQtbVicx031050poZ6oKS-7sgJD9K5ool-BRaGagu8sWNO-zKif79W_P3Cw/s1600/20130710_git_workflow_2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="132" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRzYzJB6xWMDDfi-m3nfGPJtF_BVh-wrCsjMzmHeQanGw7-JkLq7l9_UntYPGyV1ywYkc0iAyCZYMTufCyrVQtbVicx031050poZ6oKS-7sgJD9K5ool-BRaGagu8sWNO-zKif79W_P3Cw/s320/20130710_git_workflow_2.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXjIn3OLBptm9ZDqLgkKCBE0Xi99ZvRDtdP4IsH_C0xBcDC9Kqcyr5fFr6livC9jHMtNs9GudVyH94C6H-fxMSHk-U1NHfnMLUR6WSOiS9CV4HqL9my6VAYlMHIhenT8z3gAmjp6Z6FAGg/s1600/20130710_git_workflow_3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="215" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXjIn3OLBptm9ZDqLgkKCBE0Xi99ZvRDtdP4IsH_C0xBcDC9Kqcyr5fFr6livC9jHMtNs9GudVyH94C6H-fxMSHk-U1NHfnMLUR6WSOiS9CV4HqL9my6VAYlMHIhenT8z3gAmjp6Z6FAGg/s320/20130710_git_workflow_3.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdFkpaJ8d_ZLbUiuLvVP_B4_SdRxSqrQXGiRJmfSQC01k_9LsXzXvhDS2d-dieH8Yxq0PT52XGd4Zp7ZSAlmTjnZ8GoVtFbvkw3wZtTRAmLRCEO5I1c3FWdUsLVKZz68oPArIIGzltO1-9/s1600/20130710_git_workflow_4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="177" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdFkpaJ8d_ZLbUiuLvVP_B4_SdRxSqrQXGiRJmfSQC01k_9LsXzXvhDS2d-dieH8Yxq0PT52XGd4Zp7ZSAlmTjnZ8GoVtFbvkw3wZtTRAmLRCEO5I1c3FWdUsLVKZz68oPArIIGzltO1-9/s320/20130710_git_workflow_4.png" width="320" /></a></div>
<br />rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-19052037758973313322013-06-28T12:43:00.003-07:002013-07-01T09:30:51.943-07:00Engine TardToday I got an email from Engine Yard asking me to be an Engine Yard ambassador / promoter / evangelist. Given our painful experiences with EY, I said no thanks. To their credit, they asked why. So I sent them this list, and will document it here for LOLs and posterity.<br />
<br />
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
What we didn't like about EY:</div>
<ul>
<li>your pricing is unjustifiably high considering the shortcomings below.</li>
<li>your approach to database replication is facepalm-inducing. It's
explicitly designed *not* to be used to fail over. I lost count of how
many times I said WTF.</li>
<li>getting SSL to work properly with all the right headers is unnecessarily painful (stunnel).</li>
<li>there's no API to scale up and down by script--everything has to be done manually. WTF again, big time.</li>
<li>you've fixed this (I hope), but for a long time, removing instances
from an environment did not remove those from haproxy, which meant that
unless you ran the recipes manually, the app master would still be
sending traffic to instances that were either turned off or assigned to
another customer of yours, resulting in 404s and other hilarious
situations. I sang a Viking song of battle and sorrow when I realized
that's what was happening and you guys confirmed it (after I was done
laughing).</li>
<li>when adding or removing instances from your web UI, the number
of failures is greater than the number of successful changes performed.
More than half the time we would have to re-run the add or remove
process for the instances to be added successfully. This is for a
completely vanilla Rails app with a tiny number of servers, i.e. the
default / base case you guys are catering to.</li>
<li>when adding more than a couple of instances, the app master's
NIC gets flooded by requests used to provision the new instances, which
brings the entire stack down. We LOLed heartily when we figured out that
was going on. The only safe way to add instances is one or two at a
time. Given how long it takes to do that (see above re. failures), it's a
giant pain in the rear. </li>
<li>the default stack of app master + slaves is stupid. An active
app master serving traffic and SSL termination shouldn't also be a load
balancer. That's just dumb.</li>
</ul>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
To be perfectly blunt, I really
like the idea of your service, and I'm sure it's great for people to
get started with hosting, but every time we tried to do something with
our stack, it felt like EngineYard was designed and operated by amateurs
who don't have any experience running, let alone hosting, or offering
hosting for, a real web business. I wouldn't use EY even if there was 0
markup over AWS.</div>
PS: it got so bad we started calling you guys EngineTard and I drew the
attached. Note my Photoshop skills are not particularly advanced.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0pSCyIA65cJuDvCUwU5akiwQ3GXI4O6oHyJf-VkwvI2knM80gb-7SYKMZAMsr3cLbZTlIFQHd3eBQ1tConPIcvTiK1RMoit_nfqELLLIdxICle1EkzyE4oSXDYaRxH97clG04vVL4O3XU/s481/enginetard.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0pSCyIA65cJuDvCUwU5akiwQ3GXI4O6oHyJf-VkwvI2knM80gb-7SYKMZAMsr3cLbZTlIFQHd3eBQ1tConPIcvTiK1RMoit_nfqELLLIdxICle1EkzyE4oSXDYaRxH97clG04vVL4O3XU/s320/enginetard.png" width="200" /></a></div>
<br />
Update: I have to say EY has class and a sense of humor. After receiving this diatribe, they sent me a $25 Amazon gift card.<br />
<br />
Update: One thing I forgot to mention is the frequent billing errors. We cancelled our account on April 2; in May I got a bill for usage we didn't incur, and on July 1 we got a bill for snapshot storage and unused IP addresses for an account that's been closed for 3 months. Sigh.rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0tag:blogger.com,1999:blog-7042147755594594926.post-22121251792555623602013-06-19T09:12:00.001-07:002013-06-19T09:12:24.732-07:00RidiculousIf you use the words "ridiculous" or "ridiculously" as an intensive ("it's ridiculously easy") then we have nothing to say to each other.rogerhttp://www.blogger.com/profile/03011883838268327035noreply@blogger.com0