tag:blogger.com,1999:blog-8448513750975776442024-03-05T22:38:42.314-05:00Curiouser CodeSergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.comBlogger34125tag:blogger.com,1999:blog-844851375097577644.post-79430414718574305752020-05-20T02:02:00.000-04:002020-05-20T02:06:36.176-04:00Old Phone/Tablet as an Info Board: Update 3 - The elusive sunrises and sunsetsWhile I was messing up with the infoboard's bus widget, I also decided to touch up on the weather forecast (see the detailed description in the <a href="https://curiousercode.blogspot.com/2018/08/old-phonetablet-as-info-board-part-4.html">earlier post</a>) and and add a few more visual cues to it.<br />
<br />
I wanted to distinguish day and night, so that it becomes immediately clear how the weather behaves at sunset and during the morning commute, kind of like so:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7EFTB_HmTyYLJr7dU_1hgGq1sNyP9DWKVKOjN3Q3qy_4Lk52a4cEWaHIqIDHATIIB0lmqWL42y0XtZo_j5BuJqOfRFoolrgG9viL4AmEEokTXz17ToMJeIkEbeBFnKUi0ig6q-o-tVNc/s1600/newweather.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="209" data-original-width="1135" height="71" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7EFTB_HmTyYLJr7dU_1hgGq1sNyP9DWKVKOjN3Q3qy_4Lk52a4cEWaHIqIDHATIIB0lmqWL42y0XtZo_j5BuJqOfRFoolrgG9viL4AmEEokTXz17ToMJeIkEbeBFnKUi0ig6q-o-tVNc/s400/newweather.png" width="400" /></a></div>
<br />
<br />
My first attempt was very simple: I added a function <span class="mycode">eachTime()</span> that did this:<br />
<br />
<pre class="brush: js">function eachTime(){
var str = $(this).text();
var hhmm = str.split(":");
var sunrise = 6; var sunset = 22;
if (hhmm[0]>=sunset || hhmm[0] < sunrise) {
$(this).css("background","rgb(0,0,0)");
$(this).css("color","rgb(255,255,255)");
}}
</pre>
<br />
along with a single line in <span class="mycode">forecast()</span>:<br />
<pre class="brush: js">$("#rowTime").find("td").each(eachTime);
</pre>
<br />
However, this seemed a bit artificial to declare night as "anything from 22:00 till 6:00", so I wanted actual sunrise/sunset times to be calculated. Although, with my degree of precision of 1 hour, this seemed like a <a href="https://en.wikipedia.org/wiki/Sunrise_equation">relatively simple mathematical task</a>, there was no need to reinvent the wheel since PHP already has <a href="https://www.php.net/manual/en/function.date-sunrise.php"><span class="mycode">date_sunrise()</span></a> and <a href="https://www.php.net/manual/en/function.date-sunset.php"><span class="mycode">date_sunset()</span></a> functions. So I geared up a new PHP script as follows:<br />
<pre class="brush: php"><?php
$remote_dtz = new DateTimeZone('US/Eastern');
$remote_dt = new DateTime("now", $remote_dtz);
$offset = ($remote_dtz->getOffset($remote_dt))/3600;
$lat=43.6;$lon=-79.6;
echo '<html><head></head><body>';
echo '<span id="sunrise">';
echo(ceil(date_sunrise(time(),SUNFUNCS_RET_DOUBLE,$lat,$lon,90,$offset)));
echo '</span>';
echo(':'); echo '<span id="sunset">';
echo(ceil(date_sunset(time(),SUNFUNCS_RET_DOUBLE,$lat,$lon,90,$offset)));
echo '</span>';echo '</body></html>';
?>
</pre>
<br />
The only trick here is to get the correct GMT offset; I am also rounding up the result to the nearest hour. It is then very easy to load this script in another hidden <span class="mycode">iframe</span> like so:<br />
<pre class="brush:html"><iframe id="astronomy" style="visibility: hidden; height: 0px !important;" src="astro.php"> </iframe>
</pre>
<br />
and amend <span class="mycode">eachTime()</span> as follows:<br />
<br />
<pre class="brush: js">function eachTime(){
var str = $(this).text();
var hhmm = str.split(":");
var dom=$("iframe#astronomy").contents();
var sunrise = 6; var sunset = 22; //fallback
sunrise = parseInt(dom.find("#sunrise").text());
sunset = parseInt(dom.find("#sunset").text());
var commute = 8;
if (hhmm[0]>=sunset || hhmm[0] < sunrise) {
$(this).css("background","rgb(0,0,0)");
$(this).css("color","rgb(255,255,255)");
if (hhmm[0]==commute){ $(this).css("color","rgb(225,255,0)");$(this).css("border","1px solid yellow");}
}
else{ if (hhmm[0]==commute) {$(this).css("border","1px solid black"); $(this).css("background","rgb(255, 213, 171)");}}
}
</pre>
<br />
<br />
There is no need to bother about refreshing <span class="mycode">astro.php</span> because <a href="https://curiousercode.blogspot.com/2018/08/old-phonetablet-as-info-board-final.html">the entire system refreshes</a> at 2 am every day anyway, and even if we happen to be 1 day off we are well within the desired accuracy margin. In addition I have added a cue mark for the morning commute, which is (unfortunately) independent of where the Sun happens to be. <span class="myinline">(And your mileage may vary here too, distinguishing weekdays from weekends and even accounting for statutory holidays if need be.)</span><br />
<br />
<br />
<div class="myaux">
BONUS: As an addition, notably after a strong wind storm in our area, I wanted to colour code wind as well, along with rain and temperature. Here's what I changed about <span class="mycode">eachWind()</span>:<br />
<br />
<pre class="brush: js">function windgradient(base,gust){
var b=(base>=80)?1.0:(base/80.0);
var g=((gust-base)>=25)?1.0:((gust-base)/25.0);
if (g<0) g=0;
b=Math.pow(b,0.75);g=Math.pow(g,1.0);
return "rgb("
+ Math.round(255.0) + ","
+ Math.round(255.0*(1.0-b*g)) + ","
+ Math.round(255.0*(1.0-b)) + ")";
}
function eachWind(){
var rawstr = str = $(this).text().trim();
var str = rawstr.split(String.fromCharCode(160));
var result = "rgb(255,255,255)"; var speed = 0;
try { speed = parseInt(str[1]);} catch(ignore){speed=0;}
var gust=speed;
if(rawstr.indexOf("gust")!=-1) {try { gust = parseInt(str[2]);} catch(ignore){gust=speed;}}
result=windgradient(Math.round((speed+gust)/2.0),gust);
$(this).css("background",result);
$(this).text(rawstr.replace("gust",">"));
}
</pre>
<br />
<br />
So the colour intensity and hue can independently give some information about how strong and how gusty the wind is, more or less like so (yes I did it on <a href="https://www.wolframcloud.com/">Wolfram Cloud</a> for a quick illustration):<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGab4Pm7LkiTQ71YV_kPw6N-c5VnUXGeHHg4Y4Ut_3ogspB8fi4LT6z-KTyNVuNLw_J8jrtEby0_lMB8J9GWtrQJypqEXYuv_wassFt8QDjkuGI9N78dNJpm7taq-5sG6UCsuWXXXFqWE/s1600/windgradient-new.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="555" data-original-width="808" height="273" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGab4Pm7LkiTQ71YV_kPw6N-c5VnUXGeHHg4Y4Ut_3ogspB8fi4LT6z-KTyNVuNLw_J8jrtEby0_lMB8J9GWtrQJypqEXYuv_wassFt8QDjkuGI9N78dNJpm7taq-5sG6UCsuWXXXFqWE/s400/windgradient-new.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
</div>
<br />Sergeihttp://www.blogger.com/profile/02237867271347372287noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-49735093712065223152020-05-15T11:48:00.002-04:002020-05-19T09:36:08.822-04:00Old Phone/Tablet as an Info Board: Update 2 - The snappy client-side bus board updatesLast time I promised to touch upon what I did on the client side of the bus widget to make it look like this:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjW4xGT50e64wamCll6dQrdEXuLmtGW93mCLBkXfz2nIIi7lcUiyemTYcpuXiFxnkfKme9PhnzztJ05jz-Hng3zDdVMRSVeqT5WdEQWTiTr9Zf5gRWNAPyAnflGJE_PDNFS35sM02GUdbQ/s1600/shotbus2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="82" data-original-width="490" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjW4xGT50e64wamCll6dQrdEXuLmtGW93mCLBkXfz2nIIi7lcUiyemTYcpuXiFxnkfKme9PhnzztJ05jz-Hng3zDdVMRSVeqT5WdEQWTiTr9Zf5gRWNAPyAnflGJE_PDNFS35sM02GUdbQ/s1600/shotbus2.png" /></a></div>
<br />
In simple terms, I needed to solve the following problem: <i>given the timetable loaded X minutes ago, find the number of minutes remaining until the bus departures <b>now</b>, and based on that, highlight the buses that are convenient to catch (i.e. which depart roughly when I reach the bus stop if I leave soon)</i>.<br />
<br />
We remember that the timetable, as loaded, returns its output nicely compartmentalized into distinct <span class="mycode"><span></span>'s:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjn8f0NZr9ah6Owqt3Rj2cGK_KRXkogioJAzCLgoChoBlTCv7E4HpEnOAhPpOxnY84SUgOPrEYAWeJzuPq4pO2IzyLaxPa98XFy-SG115_qC13ADI-SoA-Wtk99C6Vo5-Hs5_2yLUcikjA/s1600/shotbus1ann.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="124" data-original-width="912" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjn8f0NZr9ah6Owqt3Rj2cGK_KRXkogioJAzCLgoChoBlTCv7E4HpEnOAhPpOxnY84SUgOPrEYAWeJzuPq4pO2IzyLaxPa98XFy-SG115_qC13ADI-SoA-Wtk99C6Vo5-Hs5_2yLUcikjA/s1600/shotbus1ann.png" /></a></div>
<br />
so the idea is to do the following at regular intervals:<br />
<br />
<ul>
<li>query all <span class="mycode">#bustime</span>'s for departure times;</li>
<li>determine the time difference between those times and the current time</li>
<li>update the number of minutes in <span class="mycode">#busreal</span>'s</li>
<li>re-colour all <span class="mycode">#bus*</span> according to the number of minutes left, taking into account the time it takes to walk to the bus stop.</li>
</ul>
The ideal point of insertion from the point of view of the <a href="https://curiousercode.blogspot.com/2018/08/old-phonetablet-as-info-board-final.html">flow control</a> would be the <span class="mycode">clock()</span> function, which is scheduled to run every second to update the clock; every 10 seconds I ask it to do the following:
<br/><br/>
<pre class="brush:js">
function clock(){
var now = new Date(); var h=now.getHours(); var m=now.getMinutes(); var s = now.getSeconds();
$("#nowclock").text(((h>=10)?h:("0"+h)) + ":" + ((m>=10)?m:("0"+m)) + ":" + ((s>=10)?s:("0"+s)));
if (s % 10 ==0 ) {
var mintogo = 6; // walking distance to stop in minutes
var togo = -1; var nextgo = -1; var num=0; var fnum=-1;
var dom = $("iframe#miway").contents();
while(num<6) {
var token = "#bustime"+num;
var nextbus = dom.find(token).text().trim();
nextbus.replace("now","0 min");
var am = (nextbus.indexOf('am')>0); var pm = (nextbus.indexOf('pm')>0)
var idx = nextbus.indexOf(' ');
if (idx>0) nextbus = nextbus.substr(0,idx);
var ddot = nextbus.indexOf(":"); if (ddot<0) return;
var hh = parseInt(nextbus.substr(0,ddot)); var mm = parseInt(nextbus.substr(1+ddot,nextbus.length-ddot));
if (am && hh==12) hh=0; if (pm && hh<12) hh+=12;
var hcarry=(h<20 && hh>20); var hhcarry=(h>20 && hh<20);
if (hcarry) h+=24; if (hhcarry) hh+=24;
var diffmin = (mm+hh*60) - (m+1+h*60);
togo = diffmin - mintogo;
var thiscol = buscolor(togo);
var token = "#busnum"+num; if (num==0) token="#bustoken";
dom.find(token).css("background", thiscol);
var mintxt = ':'+diffmin+'m';
if (diffmin<0) mintxt = ':-';
if (diffmin==0) mintxt = ':now';
if (num==0) {
mintxt = '('+diffmin+' min)';
if (diffmin<0) mintxt = '(--)';
if (diffmin==0) mintxt = '(now)';
}
dom.find("#bustime"+num).css("color", thiscol);
dom.find("#bustime"+num).text('\xa0'+hh+":"+((mm<10)?"0":"")+mm+'\xa0');
dom.find("#busreal"+num).css("color", thiscol);
dom.find("#busreal"+num).text(mintxt);
if (togo>=0 && nextgo<0) { nextgo=diffmin; fnum=num+1; }
num++;
}
dom.find("#realtime").text("leave in " + (nextgo-mintogo) + " min");
}}
</pre>
<br />
<br />
where the auxiliary function <span class="mycode">buscolor()</span> is simply<br />
<br />
<pre class="brush: js">
function buscolor(togo){
if (togo<0) return 'rgb(200,180,180)';
if (togo<=2) return 'rgb(255,200,0)';
if (togo<=4) return 'rgb(255,0,0)';
if (togo<=8) return 'rgb(255,64,64)';
if (togo<=15) return 'rgb(255,100,100)';
if (togo<60) return 'rgb(255,128,128)';
if (togo<120) return 'rgb(255,128,255)';
return 'rgb(255,0,255)' // error
}
</pre>
<br />
That's it. Note the following:<br />
<br />
<ul>
<li>The code handles both 12-hour and 24-hour times but the output is forced into 24-hour because it offers smaller footprint.</li>
<li>The item number 0 has a different, more verbose output format; the additional <span class="mycode">#realtime</span> contains a hint when to leave home for the "next suitable" bus.</li>
<li>The container that should be called <span class="mycode">#busnum0</span> is called <span class="mycode">#bustoken</span> instead. This is for historical reasons: the system looks for <span class="mycode">#bustoken</span> as an indicator that the timetable has loaded.</li>
</ul>
<br />
It remains to be seen whether doing this every 10 seconds will prove feasible for the Playbook in terms of "mean time between refreshes / restarts". <br />
<br />
<div class="myinline">P.S. Turns out I had planned this feature all along, according to <a href="https://curiousercode.blogspot.com/2018/08/old-phonetablet-as-info-board-part-5.html">my own original post</a>:<br /></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_JaCZYNg47kkXh7VZ99bl5l_Vo1ulmjbkQ38eSMGaWX6jfpPcAlMBTmOHjzxBV9dsBc4XWz0dDMiNDmUVSIa-XUP7w8lLiIuyHsbpTLeRFYWNy2GPyjBdNvIP397SgkXKu7O5QSgApXk/s1600/fromthepast.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="110" data-original-width="621" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_JaCZYNg47kkXh7VZ99bl5l_Vo1ulmjbkQ38eSMGaWX6jfpPcAlMBTmOHjzxBV9dsBc4XWz0dDMiNDmUVSIa-XUP7w8lLiIuyHsbpTLeRFYWNy2GPyjBdNvIP397SgkXKu7O5QSgApXk/s1600/fromthepast.png" /></a></div>
<br/>
<div class="myaux">Our mileage may vary even further. For example:<br />
<br />
<ul>
<li>we may imagine the code to throw out "missed" departures, moving the remaining ones up the queue, and even triggering an extra timetable refresh when there are too few (say <3) left;</li>
<li>we may intelligently trigger an extra refresh every time we are nearing the "leave home in 2 minutes situation in order to make sure that the upcoming bus is still on schedule;</li>
<li>we may keep track of how much the real-time departure information is changing between refreshes (provided they happen often enough) and detect when delays start to appear or frequently change (this is indicative of unstable traffic, warranting more frequent refreshes)...</li>
</ul>
...but we aren't building an after-market professional grade departure board here, and "perfect" is a very common enemy of the "good enough". </div>Sergeihttp://www.blogger.com/profile/02237867271347372287noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-28468663202203878432020-05-14T17:38:00.000-04:002020-05-15T14:32:52.024-04:00Old Phone/Tablet as an Info Board: Update 1 - The sneaky Mississauga busesSince I wrote my <a href="https://curiousercode.blogspot.com/2018/08/old-phonetablet-as-info-board-table-of.html">original series</a> 2 years ago, my custom made info board has proved a (moderately) trusty friend and a time saver, especially on busy mornings. It does need a manual refresh every now and then (roughly once every 1-2 weeks), and the <a href="https://en.wikipedia.org/wiki/BlackBerry_PlayBook">Playbook </a>itself needs a reboot once every couple of months, but this is part of regular maintenance <span class="myinline">and <i>hey, the thing is still running fine on a year-2011 Playbook and a 2013 DiskStation</i>, which is, in and of itself, a proof that I did a decent job, and an occasion to celebrate - so cheers :)</span><br />
<br />
However, nothing is forever, especially on the Internet. Over time, the board started losing components, which prompted more in-depth maintenance and code revision. So I am writing a series of update posts to list and describe that maintenance.<br />
<br />
One of the components that bit the dust lately was the departure board for the buses at our nearby bus stop (<a href="https://curiousercode.blogspot.com/2018/08/old-phonetablet-as-info-board-part-5.html">see my earlier post</a> on how I did it, I will be referencing it often.). Contrary to my fear that my brute-force string parsing of a JSON in PHP would give me debugging headaches, it actually proved remarkably robust. Something totally different happened: the query for the JSON was simply giving me <a href="https://web.mississauga.ca/miway-transit/NextPassingTimes/RequestNextPassingTimes">Error 404</a>. Apparently, the company just scrapped the service and seems to have migrated to a wholly different platform. Well, as I said, nothing on the Internet is forever, so I needed to re-implement the web scraping for the next bus departure times.<br />
<br />
I will forgo the description of a 3-hour long troubleshooting spent in Chrome's web inspector feature (mostly because it happened too long ago for me to recover the details of it); suffice it to say that I ended up finding a new request in this format:<br />
<span class="mycode">https://www.triplinx.ca/en/NextDeparture/NearByNextDeparture?stopId=(6-digit stop ID)</span><br />
resulting in the following:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuv10O1hxzIHRgS3f-WJFCAQPaDp9xUEGPwxz6zxxTE9l3d2-w1eqUDMIsx6FrfZAw1CRpYOA2yQWT5oigC73HZ8VSXrAsRHdvISBdVAwyA2bzVTUplDJ_BW312zRoNIXTe5_IkWwlhMg/s1600/new_buses.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="277" data-original-width="605" height="182" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuv10O1hxzIHRgS3f-WJFCAQPaDp9xUEGPwxz6zxxTE9l3d2-w1eqUDMIsx6FrfZAw1CRpYOA2yQWT5oigC73HZ8VSXrAsRHdvISBdVAwyA2bzVTUplDJ_BW312zRoNIXTe5_IkWwlhMg/s400/new_buses.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
Sweet. A simple Inspect shows a clear document structure:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEii0rKaIQw-94ZeXdXCjBwkFYbcowPtU2A53Jidnt24Yc8qTf4kPNUjlOXVit9hEnR-LHLh5k6F-KOLgQJw_QsqkYranrjSwIRavfABmveHDm4iqR3ZmNgkejCKikBrxU3uzpsp5rEQ0xs/s1600/new_buses2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="295" data-original-width="729" height="161" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEii0rKaIQw-94ZeXdXCjBwkFYbcowPtU2A53Jidnt24Yc8qTf4kPNUjlOXVit9hEnR-LHLh5k6F-KOLgQJw_QsqkYranrjSwIRavfABmveHDm4iqR3ZmNgkejCKikBrxU3uzpsp5rEQ0xs/s400/new_buses2.png" width="400" /></a></div>
<br />
I wasn't inclined to modify my "<a href="https://curiousercode.blogspot.com/2018/08/old-phonetablet-as-info-board-part-5.html">bake the page directly in PHP</a>" method, but I could now afford to parse the document somewhat more intelligently, like so:<br />
<br />
<pre class="brush: php">
function getElementsByClass(&$parentNode, $tagName, $className) {
$nodes=array();
$childNodeList = $parentNode->getElementsByTagName($tagName);
for ($i = 0; $i < $childNodeList->length; $i++) {
$temp = $childNodeList->item($i);
if (stripos($temp->getAttribute('class'), $className) !== false) {
$nodes[]=$temp;
}
}
return $nodes;
}
$url = "https://www.triplinx.ca/en/NextDeparture/NearByNextDeparture?stopId=$stop";
$code = file_get_contents($url,false,$context);
$routes=array(); $times=array();$rid=0;
$dom = new DOMDocument();
@$dom->loadHTML($code);
$list_node=$dom->getElementsByTagName("ul")->item(0);
$items=$list_node->getElementsByTagName("li");
for($i=0;$i<$items->length;$i++)
{
$thisraw = $items->item($i)->nodeValue;
// set route - we are in the outer LI
$token1 = "Bus"; $pos1 = strpos($thisraw,$token1)+strlen($token1)+2;
$token2 = "Show"; $pos2 = strpos($thisraw,$token2)-2;
$rawroute = substr($thisraw,$pos1,$pos2-$pos1);
if (strlen($rawroute) <= 5)
{$thisroute=$rawroute; }
else // we are in the inner LI, so get time for the already set route
{
$thisattr=0; $thisattr += (strpos($thisraw,"real time")!==FALSE) ? 1 : 0; // real time attr
$attr[] = $thisattr;
$diritem = getElementsByClass($items->item($i),"span","next-departure-label")[0];
$direction = $diritem->nodeValue;
$direction = str_replace("towards ","", $direction);
$routes[] = $thisroute . substr($direction,1,1) . " ";
$timitem = getElementsByClass($items->item($i),"span","next-departure-duration")[1];
$thistime = $timitem->nodeValue;
$thistime = str_replace("<", "",$thistime); // filter < 1 minute:: gone anyways
$thistime = str_replace("<", "",$thistime);
$entity = htmlentities($thistime, null, 'utf-8');
$thistime = str_replace(" ", " ", $entity);
$thistime = html_entity_decode($thistime);
$times[] = $thistime;
$rid++;
}
}
</pre>
<br />
where I borrowed the function <span class="mycode">getElementsByClass()</span> <a href="https://stackoverflow.com/a/31616848">from StackOverflow</a>; the snippet aimed at getting rid of all <span class="mycode">&nbsp;</span>'s was likewise <a href="https://stackoverflow.com/a/21801444">borrowed from there</a>.<br />
<br />
As a result I was now getting an array of bus routes and bus times in string arrays <span class="mycode">$routes[]</span> and <span class="mycode">$times[]</span> respectively. However, I still needed to sort my buses by departure time rather than by route as in the screenshot above; the trick is that some of the times are given as "5 min" while some others are like "2:35 pm".<br />
<br />
I got around the problem by determining the "number of minutes till all departures" and storing it in <span class="mycode">$nummin[]</span>, like so<br />
<br />
<pre class="brush:php">
date_default_timezone_set('US/Eastern');
$rawtime = new DateTime();
$rid = (($rid>6)?6:$rid);
for ($j=0; $j<$rid; $j++)
{
if (strpos($times[$j],"min")!==false)
{
$nummin[$j] = intval(str_replace("min","",$times[$j]));
}
else
{
$temptime = new DateTime($times[$j]);
$fulltime = getdate($temptime->getTimestamp());
$temptime->setDate($fulltime['year'],$fulltime['mon'],$fulltime['mday']);
$nummin[$j] = intval(($temptime->getTimestamp() - $rawtime->getTimestamp())/60);
if($nummin[$j]<0)$nummin[$j]+=(24*60); // add a day if needed
}
}
</pre>
<br />
and then using it to bubble sort all three arrays (6 elements aren't worth doing anything more sophisticated):<br />
<br />
<pre class="brush:php">
for ($i=0; $i<$rid; $i++)
for ($j=0; $j<$i; $j++)
if($nummin[$i]<$nummin[$j])
{
$temproute = $routes[$i]; $routes[$i] = $routes[$j]; $routes[$j] = $temproute;
$temptime = $times[$i]; $times[$i] = $times[$j]; $times[$j] = $temptime;
$tempmin = $nummin[$i]; $nummin[$i] = $nummin[$j]; $nummin[$j] = $tempmin;
}
</pre>
<br />
For output, we would need the reverse operation, i.e. converting "in 5 minutes" to a valid departure time. The reason is that we can't afford to pull the timetable every single minute, so labels like "in 5 min" would very soon mean "in 5 minutes, as of 3 minutes ago" which isn't very convenient to use. To get around this confusion, I have employed the following trick:<br />
<br />
<pre class="brush:php">
for ($j=0; $j<$rid; $j++)
{
if (strpos($times[$j],"min")!==false)
{
$live = $times[$j];
$mins = intval(substr($live,0,strpos($live," min")));
$newtime = (clone $rawtime);
$newtime->modify("+{$mins} minutes");
$bustime = date("H:i",$newtime->getTimeStamp());
$times[$j] = $bustime . " (" . $live . ")";
}
}
if (strlen($times[0])<4) $times[0]= $timestamp." (now)";
</pre>
<br />
<div class="myaux">
Note the line with <span class="mycode">$newtime = clone $rawtime</span>. It is very important that <span class="mycode">$newtime</span> is cloned, otherwise <span class="mycode">$newtime->modify(...)</span> will modify <span class="mycode">$rawtime</span> and our code will work, but will produce a wrong timetable! Also note that the last line is a patch to ensure that "<1 min" is captured as "now" regardless of possible parsing errors upstream (spoiler: there are errors upstream).</div><br />
<br />
As an afterthought, as I have both the number of minutes and time for all departures, I decided to output them both, breaking up the elements and giving them distinct IDs for future access. Here's how:<br />
<pre class="brush:php">
$css= ' style="font-weight: bold; color:rgb(255,128,128);"';
for ($j=0; $j<$rid; $j++)
{
if (strpos($times[$j],"(")!==false)
{$times[$j] = str_replace('(','</span><span id="busreal'.$j.'" '.$css.'>(',$times[$j]); }
else { $times[$j] = $times[$j].'</span> <span id="busreal'.$j.'" '.$css.'>('.$nummin[$j].' min)';}
if ($j==0) $css=str_replace(';"','; font-size: 14px;"',$css);
}
. . .
echo '<span id="bustoken" style="background: ',$color,'; color:white;">', " ".$routes[0], '</span>', '<span id="bustime0" style="font-weight: bold; color:',$color,'">', " ", $times[0], '</span>', ' <span id="realtime" style="color: ',$color,'; ">[...]</span>';
if ($rid > 1)
{echo "<br/>";
echo '<span style="color:rgb(255,235,235);">', $timestamp, ' </span>';
for ($j=1; $j<$rid; $j++)
{echo '<span id="busnum'.$j.'" style="background: rgb(255,128,128); color:white; font-size: 14px;">', " ".$routes[$j], '</span>', '<span
id="bustime',$j,'" style="font-weight: bold; color: rgb(255,128,128);font-size: 14px;">', " ", $times[$j], '</span> ';
};
}
</pre>
<br />
<div class="myinline">You can surely notice that the "breaking up" thing was really an afterthought and is not neat code at all. I will probably yell at myself for doing it this ugly when I decide to refactor this in another 2 years. :)</div><br />
<br />
So, here is the final result:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNV6nle2RXTtOrMfV_LsigX6ZVL4R8Oj2WWimndBwSqAtZdTkMRwXUUlCkSqfsTTIug74fLgShuoxdqoPvjuUPU5OlcHa3GIFYHC3bVrvlmm-KjLWI1v9tWZHJEeLJEWxOpf_ZVlBUU7U/s1600/shotbus1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="62" data-original-width="912" height="43" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNV6nle2RXTtOrMfV_LsigX6ZVL4R8Oj2WWimndBwSqAtZdTkMRwXUUlCkSqfsTTIug74fLgShuoxdqoPvjuUPU5OlcHa3GIFYHC3bVrvlmm-KjLWI1v9tWZHJEeLJEWxOpf_ZVlBUU7U/s640/shotbus1.png" width="640" /></a></div>
<br />
And this is how it looks like in an embedded form:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgr45ApOIcsRX7IgdNMj_dqP0aNnBWmJt0gxNk9hxfVrfB-oWHbRVWc9WTgHzEcIVbftjTiuO00cMCe_m3ve6bZ19YYcxWu0T16JWg8uqNWB4sOIg5IWxv0LXKEcJjg90G-dLZlerawDak/s1600/shotbus2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="82" data-original-width="490" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgr45ApOIcsRX7IgdNMj_dqP0aNnBWmJt0gxNk9hxfVrfB-oWHbRVWc9WTgHzEcIVbftjTiuO00cMCe_m3ve6bZ19YYcxWu0T16JWg8uqNWB4sOIg5IWxv0LXKEcJjg90G-dLZlerawDak/s1600/shotbus2.png" /></a></div>
<br />
<br />
<div class="myaux">What's with all the different colours, you ask, what does "leave in..." mean, and why is formatting so different? I bet you guessed it: the 16:31 bus is already too soon to catch, and the 16:33 can be caught just barely, if you leave now and make haste. In my next post, I am going to describe how I achieved this on the client side.</div><br />
Sergeihttp://www.blogger.com/profile/02237867271347372287noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-51264535549808191902020-05-10T12:15:00.000-04:002020-05-14T13:10:54.388-04:00Computer surgery, DVD player laparoscopy, and even more debugging without a debugger<h3>
Pre-prologue: Computer surgery</h3>
Long story short: the battery on my old Mac gave up the ghost in a relatively ugly way and I had to put a bit of effort for that ghost not to become a fire spirit. In one picture worth a thousand words, this is what happened:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaiUrvpe3wkWTezEtH3Du0FzlSiiKyjtQN1HhxE66wHclOj-c0XFvhJzSj4xgfzDJP3Z1PyZlbLM33PmWqNGHPBmlXnUV59J66zJkycHVMoe1DcHkfdoOs4iNhzVobgodwFUoDxtNUD8k/s1600/foto_no_exif-combo.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="499" data-original-width="1600" height="196" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaiUrvpe3wkWTezEtH3Du0FzlSiiKyjtQN1HhxE66wHclOj-c0XFvhJzSj4xgfzDJP3Z1PyZlbLM33PmWqNGHPBmlXnUV59J66zJkycHVMoe1DcHkfdoOs4iNhzVobgodwFUoDxtNUD8k/s640/foto_no_exif-combo.jpg" width="640" /></a></div>
<br />
Which meant that the only computer with a DVD drive was severely crippled, and I was getting fed up with hooking a computer to the TV whenever there was a "please Daddy may we watch..." So I decided to buy a simple standalone <a href="https://www.amazon.ca/LG-BP350-Blu-Ray-Disc-Player/dp/B01AWMVI9M">DVD/Blu-ray player</a>.<br />
<br />
<h3>
Prologue: DVD players and world travellers</h3>
The trouble was, I have lived on 2 continents in 5 different countries (at one time, simultaneously), and while doing so, a hodge-podge mini collection of DVDs from at least 3 different regions materialized on my living room shelf. While I fully understand the region mechanism so that a new blockbuster may hit the box offices at different times in different parts of the world, this protection kind of makes zero sense for titles (or worse, indies) released around the world long ago. Of course there's <a href="https://www.videolan.org/">VLC</a> and <a href="https://www.videolan.org/developers/libdvdcss.html">dvdlibcss</a> but hey, this whole thing was done to spare the hassle of having to connect a computer to the TV to play a disc.<br />
<br />
In other words, I needed a region freeing or region switching mechanism for my player. I researched some forums before buying, and it seemed that at least *some* players of this kind could play discs from around the world; also there were already "multi-region" players of this kind freely available for sale <a href="https://www.amazon.ca/MultiZone-012345678-Playback-Support-World-Wide/dp/B01NAAETXP">on Amazon</a> and <a href="http://www.codefreedvd.com/region-free-blu-ray-players/lg-bp350-region-free-blu-ray-player.html">third-party websites</a> (presumably with modified firmware), so I thought that it might be worth to try my luck as well.<br />
<br />
Expectedly but disappointingly, the unit was not multi-region out of the box. None of the methods "off the internet" worked either. None of the <a href="https://www.directutor.com/content/how-remove-region-codes-dvd-or-blu-ray-players">"magic" remote control codes</a>. Some more in-depth sites suggested <a href="https://www.videohelp.com/dvdhacks/lg-bp440/12388">burning a special CD-R</a> to enter the "advanced settings mode", but this did not work for me at all (and people were writing in the forum that this CD-R method was good for any player except North American, which was my case). The forums even <a href="https://www.blogger.com/"><span id="goog_940390750"></span>mentioned</a> that the only way was to physically get into the player by soldering some wires to its serial interface <span class="myinline">(yikes: I am not that big a fan of soldering and my <a href="https://curiousercode.blogspot.com/search/label/electronics">few circuits</a> even tend to work better on a breadboard than in soldered form)</span>, so it seemed that I had to use the computer route for anything outside North America...<br />
<br />
<h3>
Entry route and initial troubleshooting</h3>
...or not? Further search got me to <a href="https://www.exploitee.rs/index.php/LG_BP350%E2%80%8B%E2%80%8B">a page</a> mentioning what is called "the Pandora exploit". Here's the essence of it:<br />
<br />
<pre class="brush: bash" style="font-size: smaller;"><code>$ cat /mnt/rootfs_normal/usr/local/bin/pandora/pandora.sh
#!/bin/sh
#
(...)
if [ -e /mnt/sda1/PandoraApp ]; then
(...)
/mnt/sda1/PandoraApp -qws -display directfb
elif [ -e /mnt/sdb1/PandoraApp ]; then
(...)
/mnt/sdb1/PandoraApp -qws -display directfb
else
(...)
/usr/local/bin/pandora/PandoraApp -qws -display directfb
fi
</pre></code>
What this means is that when the player is asked to launch the Pandora app (I had no idea what this was, at the time), the player looks for <span class="mycode">PandoraApp</span> executable in the following directories, <i>in that order</i>: <span class="mycode">/mnt/sda1/</span>, <span class="mycode">/mnt/sdb1/</span>, and finally <span class="mycode">/usr/local/bin/</span>. Which is to say that if you put a shell script called <span class="mycode">PandoraApp</span> in the root folder of a USB stick, plug it into your player, and launch Pandora, it will execute that script instead of the app.<br />
<br />
This was worth trying out. Pandora isn't available in Canada so there was no menu item to call it, but as soon as I changed the player's country to US, that menu item happily appeared in the Premium menu of the player (which holds all the "streaming app" capabilities). Provided the player was connected to the Internet (i.e. had wi-fi set up and running), I was able to get to a nice splash screen where I was <a href="https://www.pandora.com/restricted">happily informed</a> that, well, "Pandora isn't available in your country".<br />
<br />
The exploit page mentioned setting up a reverse shell, but this seemed to me too complicated in a home network situation full of dynamic IPs and firewalls. Instead, what I put on the SD card was<br />
<br />
<div class="mycode">
ls > /mnt/sda1/check_a.txt; ls > /mnt/sdb1/check_b.txt;</div>
<br />
This time, running Pandora resulted in the USB stick LED blinking once and then the player froze with a black screen, forcing me to do a hard reboot. The USB stick nicely contained the file <span class="mycode">check_a.txt</span>, holding root directory listing, and confirming that the drive was indeed mounting under <span class="mycode">/mnt/sda1</span>.<br />
<br />
Next time, what I put on the drive was:<br />
<br />
<div class="mycode">
ls -alR > /mnt/sda1/dirlist ;<br />
/usr/local/bin/pandora/PandoraApp -qws -display directfb;</div>
<br />
The last line ensured that the Pandora app was launched after what I instructed the player to do, allowing me to return to the main menu and avoiding the need for a hard reset. After a few unsuccessful tries, I got a nice full listing of the player's file system.<br />
<br />
<div class="myaux">
The reason for "a few tries" turned out to be that I was accidentally using a defective USB drive which, due to its old age or/and compatibility issues, had a corrupted file system that took the player forever to list. So, kind of "bad luck". However, I had a compensating "good luck" that the first stick I tried ended up working - the newer ones did not, even the first script wasn't being executed at all, presumably because newer drives were mounted under some different path. Had I started with a non-working stick, I might well have thought that the exploit didn't work anymore, most likely got patched in the newer firmware, and would have given up any further tries... the lesson of this was not to give up and "throw it until it sticks" a few more times.</div>
<br />
<br />
<h3>
Analysis and solution</h3>
Okay, I have the full listing of the player's system, now what?
<br />
<br />
I spent some time carefully studying the listing, and after a few red herrings stumbled onto this:
<br />
<br />
<div class="mycode">
./mnt/ubi_boot/var/local/acfg:<br />
total 108<br />
drwxr-xr-x 2 root root 232 Jun 1 2014 .<br />
drwxr-xr-x 6 root root 424 Jan 1 1970 ..<br />
-rw-r----- 1 root root 107677 Jun 1 2014 config_file.txt
</div>
<br />
So copying the file to the USB stick for investigation by doing<br />
<br />
<div class="mycode">
cp -f -v /mnt/ubi_boot/var/local/acfg/config_file.txt /mnt/sda1/ ;<br />
/usr/local/bin/pandora/PandoraApp -qws -display directfb;</div>
<br />
and opening the copied file in a hex viewer such as <a href="https://en.wikipedia.org/wiki/Hiew">hiew</a> immediately gave me this;<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6h0z9HD0nx5ZAOxI9VFXJQnbbvMb4dkffpaYBH8OcskkseAXux3nCwWuGdxwi0QnvA-8SKZHzSYz2kIFx2t9Q1TLfQmpwx4I7TQdkzCFKnhml6jOH0gZb5X9AzIuF0Ur3tzMs02eeOr4/s1600/hiew-1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="638" data-original-width="1102" height="185" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6h0z9HD0nx5ZAOxI9VFXJQnbbvMb4dkffpaYBH8OcskkseAXux3nCwWuGdxwi0QnvA-8SKZHzSYz2kIFx2t9Q1TLfQmpwx4I7TQdkzCFKnhml6jOH0gZb5X9AzIuF0Ur3tzMs02eeOr4/s320/hiew-1.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
Promising. Let us zoom in for a closer look:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguoeVlmo0NMdnf_5nzLOHBjcfEfWdMQIVPyHsovBr2Z5oUlJtHjJTfwqeY05-viASC56FP_cmJG-iltDkHGO8_a6ntcb8oT6szZpUogA4AHhKeHr4PhfMxYKaQQt1Ii0XIUHIwPyL6ISc/s1600/hiew-2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="609" data-original-width="722" height="336" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguoeVlmo0NMdnf_5nzLOHBjcfEfWdMQIVPyHsovBr2Z5oUlJtHjJTfwqeY05-viASC56FP_cmJG-iltDkHGO8_a6ntcb8oT6szZpUogA4AHhKeHr4PhfMxYKaQQt1Ii0XIUHIwPyL6ISc/s400/hiew-2.png" width="400" /></a></div>
<br />
Looking at the byte signature after each option, it seemed like the first non-zero byte marked something like the beginning of the data section (always <span class="mycode">01</span>); the second marked the length of the data (<span class="mycode">01</span> for one byte, <span class="mycode">02</span> for two bytes etc.), and finally the last byte stored the relevant information (e.g. <span class="mycode">FF</span> at <span class="mycode">0x50C</span> stands for 255 years Blu-ray age restriction).<br />
<br />
At this point many would say "Got it!" and rush to change the bytes at <span class="mycode">0x3E3</span> and maybe <span class="mycode">0x408</span> from <span class="mycode">01</span> to <span class="mycode">00</span>... well, <i>not so fast</i>. Remember the exploit only works when you have a bootable player! The file is not encrypted, that much we <i>can</i> see, but what if there is a checksum somewhere that needs to match with any edit of the file? A mismatch may, at best, trigger a factory reset, and at worst, brick the player. So I changed an option from the Settings screen, for example the above mentioned age restriction to, like, 128 years. Re-dump the file, and...<br />
<br />
<div class="mycode">
<b>></b>fc /b config_file.txt config_file1.txt<br />
<br />
Comparing files config_file1.txt and config_file.txt
0000050C: 80 FF
</div>
<br />
OK, we are safe at least on this front.
<br />
<div>
For editing, you could either <a href="http://www.hiew.ru/index.html#hiew">buy the full version of Hiew</a>, or use <a href="http://www.hiew.ru/index.html#recordman">Recordman</a> which is from the same author. Here goes, for example, </div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWjxqNp2FVFfCsYLClyPZOJuLo5OdrnTo1rKRyRHH3-wC_ttR_7Ecc77R9jFvWByruLqoPcSm0RUtcDb_uDFQE-feIefBlAR-qZMexH70oKSrrrV0ahsrFz-TVprveqgvgX0Qgow7Yzio/s1600/hiew-3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="731" data-original-width="835" height="350" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWjxqNp2FVFfCsYLClyPZOJuLo5OdrnTo1rKRyRHH3-wC_ttR_7Ecc77R9jFvWByruLqoPcSm0RUtcDb_uDFQE-feIefBlAR-qZMexH70oKSrrrV0ahsrFz-TVprveqgvgX0Qgow7Yzio/s400/hiew-3.png" width="400" /></a></div>
<div>
<br /></div>
<div>
The modification of ...<span class="mycode">_BDAGE</span> is needed for us to be able to visually confirm that the new file has taken effect; the modification of ...<span class="mycode">_REGIONFREE</span> is just for good measure, I have no idea what it does. Store it as <span class="mycode">config_file_fix.txt</span>, and do the following...</div>
<br />
<br />
<div class="mycode">
cp -f -v /mnt/ubi_boot/var/local/acfg/config_file.txt /mnt/sda1/config_file_pre.txt >/mnt/sda1/report.txt ;<br />
cp -f -v /mnt/sda1/config_file_fix.txt /mnt/ubi_boot/var/local/acfg/config_file.txt >>/mnt/sda1/report.txt ;<br />
cp -f -v /mnt/ubi_boot/var/local/acfg/config_file.txt /mnt/sda1/config_file_post.txt >>/mnt/sda1/report.txt ;<br />
/usr/local/bin/pandora/PandoraApp -qws -display directfb;</div>
<br />
... to find that it does not work, and the edits revert back upon player restart.<br />
<br />
Well, after some more probing, reading, dumping, and more reading up on <a href="http://www.linux-mtd.infradead.org/faq/ubi.html">UBI</a>, <a href="http://www.linux-mtd.infradead.org/doc/general.html">MTD</a> and <a href="https://en.wikipedia.org/wiki/Flash_memory#Distinction_between_NOR_and_NAND_flash">NAND flash</a>, it occurred to me that write-behind cache might have been playing tricks on me; after all, the system totally doesn't expect a file modification at this point. So I tried adding <span class="mycode">/bin/sync</span> after copying the file, hard-reset the player some time after the script executed, and it worked; looking at settings confirmed that the BD age restriction went down from 255 to 254 years.<br />
<br />
<div class="myaux">
I have to give you a "proceed at your own risk" warning here, for two unrelated reasons. First off, I have no idea how robust the player software is against false moves such as accidentally erasing yor config file entirely or messing up in any other way; having <i>actually done</i> <span class="mycode">sudo chmod -r /</span> on my first Mac (followed by <span class="mycode">sudo chmod +r /</span> ... <i>what does it mean "sudo not found"?!?</i>) I can say that <b>it is entirely possible to brick your player this way</b>, and there will be no way back unless you really want to (and can) do the soldering thing. The other reason is that media industry does not pat us on the back for reverse engineering their code because 9 times out of 10 people try this with some form of piracy in mind. Well that wasn't my motive whatsoever (since I already had quite a few ways to play all of my entirely legal discs), <b>and it should not be your motive either</b>. So, don't try this at home unless you really know what you are doing, and if you do, <b>proceed at your own risk</b> and treat it like I treated it -- as a troubleshooting challenge (which I love) along with a comfort improving thing (which I value).
</div>
Sergeihttp://www.blogger.com/profile/02237867271347372287noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-56228315629252632752020-01-27T13:44:00.000-05:002020-01-31T14:37:02.341-05:003D printer: First impressions and a DIY fume extractor<h3>
Backstory: 3D printer first impressions <span style="font-weight: normal;">(pun intended)</span></h3>
<div>
I need to do quite a lot of round-the-house repairs, and have always fancied a 3D printer to be able to churn out custom plastic parts for replacements and custom projects. Well, as a recent birthday gift from my wife, I <a href="https://www.amazon.ca/Comgrow-Creality-Printer-Tempered-220x220x250mm/dp/B07V4RTYQF/">got one</a> - and it proved to be one of the most versatile and good-value-for-money model on the market, thanks to a very wide community support. The learning curve was a bit steep at times but it turned out I got extremely lucky and managed to get my printer going in relatively no time without causing it major damage -- although I came quite dangerously close to it more than once, I even managed to master <a href="https://all3dp.com/2/ender-3-bed-leveling-all-you-need-to-know/">bed leveling</a> without major scratches on said bed :).<br />
<br />
<div class="myaux">An accidental find: it is more reliable to <b>not</b> disable steppers for bed leveling following auto homing (moving X/Y using the control panel instead). It's more cumbersome but maintaining holding current on the Z-motor ensures that the nozzle remains exactly at the same height at the beginning of the print as it was during leveling. An even better option would be to only disable X/Y steppers while holding the Z-stepper. I may end up programming some kind of manual bed leveling wizard into the firmware at some later point.<br />
<br />
Another accidental find: it is better to <b>not</b> bundle the Bowden tube together with the hotend cables. It came bundled and I undid it by pure accident, then connected it, quite absent-mindedly, to the extruder (barely realizing that this is actually where the filament is going through), and only then realized that I cannot easily undo the tube connection to re-route it through the cable ties. So I just left it like that, and discovered that it makes a more natural filament path and puts less strain on the print head.
</div><br />
<br />
First impressions (pun intended) came out quite acceptably good, at least for my purposes (I wasn't going to set any world records in print quality or detail level or display beauty or anything line that - I simply want my parts to come out <i>functional</i>). Since I wanted to keep the printer busy for some time, I was mainly printing some more common accessories for the printer itself. <span class="myinline">(Yes, this kind of a 3D printer is very modular, like Lego for adults -- you can swap out and upgrade pretty much any part of it, and many upgrades come in the form of "download a model and print it yourself", so I outfitted mine with a <a href="https://www.thingiverse.com/thing:2917932">filament guide</a> and <a href="https://www.thingiverse.com/thing:2440593">cleaner</a>, <a href="https://www.thingiverse.com/thing:3176144">extruder knob</a>, <a href="https://www.thingiverse.com/thing:3155772">fan cover</a>, <a href="https://www.thingiverse.com/thing:3515986">tool holder</a> (a LONG almost-overnight project), etc.)</span><br />
<br />
Then came the first "real-life" application: basically a custom-sized brick of plastic with a screw hole inside to serve as a spacer for mounting a hanging file frame inside a bedroom cabinet (as part of document organization project). This was an extremely simple model that came out as desired on the second try.<br />
<br />
Then came the second "real-life" application: a holder for a <a href="https://www.thetileapp.com/en-us/store/tiles/sticker">Tile Sticker</a> that would fit on the strap of a <a href="https://www.amazon.ca/Summer-Infant-Babble-Wearable-Monitor/dp/B016Q0YY9U">Babble Belt baby monitor</a>. Thanks to Pavel from <a href="https://dotsandbrackets.com/3d-printer-tuning-and-design/">Dots and Brackets</a> who introduced me to the genius idea of <a href="https://www.openscad.org/">OpenSCAD</a> (basically a programming language for 3D modeling), the total time from concept to printout was less that 3 hours and fit nearly perfectly.<br />
<br />
I was ready to start conquering the 3D printing world but...</div><br/><br/>
<h3>
Fumes issue and extractor hood idea</h3>
<div>
...but the layout of our house left no place for the 3D printer installation other than in the basement room that doubled as gym / playroom, as well as contained one of the main HVAC returns. This basically meant that I couldn't print with anything "smelly" like ABS or, well, most materials other than PLA. Even with PLA (biodegradable / biocompatible as they claim it is) I would be a bit worried about the amount of fumes and fine particles that my prints might be generating. Especially with kids playing in the same room and the HVAC system happily blowing the basement air throughout the whole house.<br />
<br />
So in the long run, I would totally need some kind of exhaust to vent all those pesky fumes <strike>the hell</strike> out of my house. I noticed that I had an existing 4-inch dryer exhaust in the workshop room (a small 1x2 meter closet in the far corner of the basement). Immediately I thought that I could gear up something like this (additionally outfitting the workshop itself with a functional exhaust which isn't a bad idea either). After googling around for what components I can buy cheaply I came up with the following:</div>
<div>
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyP0SSGbSkUx_QHKxwS2C71oKSnMoHFgdKnY2Sf3OgQiVD6cmWxlVXK1QOhMA7GcdI21kABSpsu0wlSuCW5ysT-Ltk_qXkQ6VxhDw1jK8KlGH0hCgr8uh_t8GnAheeNGqxiulIFZdyL4Q/s1600/ductwork.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="425" data-original-width="502" height="337" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyP0SSGbSkUx_QHKxwS2C71oKSnMoHFgdKnY2Sf3OgQiVD6cmWxlVXK1QOhMA7GcdI21kABSpsu0wlSuCW5ysT-Ltk_qXkQ6VxhDw1jK8KlGH0hCgr8uh_t8GnAheeNGqxiulIFZdyL4Q/s400/ductwork.PNG" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">From right to left: L, louver; G, grille (3D printed); ED, existing duct; BD, backdraft damper; R, 6-to-4 inch reducer; Y, 6x4x4 inch Y-connector; BG, blast gate; FD, flexible duct; V, adjustable round vent</td></tr>
</tbody></table>
<br /></div>
<div>
<h3>
Buyer's list</h3>
</div>
<div>
So here was my buyer's list (all prices are in Canadian dollars). <span class="myinline">Note that I got all items off Amazon with Prime delivery, which may have increased the prices compared to Ebay or Aliexpress, but I wanted to get the work started ASAP, not in a few months:</span></div>
<div>
<ul>
<li>The hood: a <a href="https://www.amazon.ca/gp/product/B01EY1NCM8/">mini dust hood</a> ($17), as well as any suitable sheet to extend the hood to dimensions - I used an <a href="https://www.amazon.ca/gp/product/B0049MWXM8/">ABS sheet</a> I happened to have lying around ($13) but you can use literally anything you have on hand, <strike>including cardboard from an Amazon shipping box</strike>. <span class="myinline">(Well, not quite, as it turns out. The sheet needs to be strong enough to support the curtain weight, yer lightweight enough to not bring down your mount. My ABS sheet did flex a bit so may need to replace it or to print some reinforcement beams to support it.)</span></li>
<li>The fan: I used a <a href="https://www.amazon.ca/gp/product/B005KMTYFK/">6-inch inline</a> 250 CFM lying around ($35 a few years ago) but you could get a 4-inch 100 CFM inline fan for about $24. </li>
<li>The ductwork: <a href="https://www.amazon.ca/POWERTEC-70108-4-Inch-Vacuum-Collector/dp/B005VRPQ92/">blast gate</a> ($9), <a href="https://www.amazon.ca/gp/product/B01MRDYFJV/">backdraft damper</a> ($13), some <a href="https://www.amazon.ca/gp/product/B00009W3H3/">4-in flexible duct</a> ($5 for 5ft), <a href="https://www.amazon.ca/Active-Air-ACC4-Stainless-Clamps/dp/B007ISWJHO/">clamps</a> ($2 for 2), and <a href="https://www.amazon.ca/gp/product/B001NPANGW/">connectors</a> ($3.5 each). Since I used a 6-in fan, I also needed a <a href="https://www.amazon.ca/gp/product/B06XC86BLL/">6-to-4 in reducer</a> ($14), and because I designed the exhaust to also serve my workshop, I added on a <a href="https://www.amazon.ca/gp/product/B003NE5A82/">6x4x4 Y-connector</a> ($9) and an <a href="https://www.amazon.ca/gp/product/B07JLQ17R4/">adjustable 4-in round vent register</a> ($13). If you decide to merge your exhaust with your existing dryer ductwork, you will instead something like a <a href="https://www.amazon.ca/gp/product/B005VRJWGA/">4x4x4 T-connector</a> ($8.5) and another blast gate. <span class="myinline">You will also need to make sure your backdraft damper is between your fan and the insertion point to your dryer ductwork, and would need to have at least some kind of <a href="https://www.amazon.ca/Dundas-Jafine-PCLT4WZW-Dryer-Duct/dp/B01LVYZEQZ/">lint trap</a> or screen, compensated for by a stronger fan, so that your dryer will not rain lint onto your 3D printer. Ideally, you'd want some sensor lock-out preventing your dryer from running when the blast gate to your printer is opened, otherwise your filament won't like the warm moist air coming from your dryer.</span> </li>
<li>The mount. I wanted it to be flexible so that I could raise my exhaust hood when not in use, therefore this part proved the most problematic. A <a href="https://www.amazon.ca/gp/product/B07ZJ3QX29/">gooseneck phone holder</a> didn't work (way too limp); an <a href="https://www.amazon.ca/gp/product/B07TWYHCZH/">adjustable screen arm</a> wasn't adequate either (too heavy, not enough articulation in the needed directions, hot tight enough in other directions). But microphone mounting equipment proved <strike>quite fine</strike> just barely adequate (I ended up getting a <a href="https://www.amazon.ca/gp/product/B0002E54ZU/">boom arm</a>, a <a href="https://www.amazon.ca/gp/product/B00080KNH2/">flange</a>, a <a href="https://www.amazon.ca/gp/product/B00080LUJW/">short gooseneck</a> and a <a href="https://www.amazon.ca/gp/product/B01M1FHTPS/">set of nuts</a> for a total of $41, roughly the price of a monitor arm.)</li>
<li>The curtains. I reused a clear plastic table covering sheet outfitting it with an abundance of <a href="https://www.amazon.ca/gp/product/B07GBWTGNX/">Velcro strips</a> ($10 for 5m). More on this later.</li>
</ul>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhwZp6m8ShRHj3qJHsEEb4MmgdPUyTly4GvWN2Uvf0E_MzOY1ofqkbWd4o8CynPe4DY2DjPuL5lmPPuveiQuqWug51M0tKHn59s77gmUrC3qjfqiprPs1VPeF0oys84B9Vrmldx7FG49U/s1600/parts.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1140" data-original-width="1600" height="228" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhwZp6m8ShRHj3qJHsEEb4MmgdPUyTly4GvWN2Uvf0E_MzOY1ofqkbWd4o8CynPe4DY2DjPuL5lmPPuveiQuqWug51M0tKHn59s77gmUrC3qjfqiprPs1VPeF0oys84B9Vrmldx7FG49U/s320/parts.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Some of the parts for the project (a few of them were returned and upgraded).</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
<div>
<ul>
</ul>
</div>
</div>
<div>
<h3>
Implementation</h3>
</div>
<h4>
0. Electrical wiring for the fan</h4>
<div>
This step was optional but I felt the need to do it because I had no readily available wall outlets near the fan, and did not want to mess with extension cords. So I ended up rummaging my electrical supplies box for a standard 2-outlet receptacle, a Decora switch, two 2x4 electrical boxes, two outlet covers, several oddball clamp connectors, and a bit of 14/3 cable to put in a switch-controlled plug for the fan(s). </div>
<div>
<br /></div>
<div>
I decided to branch the circuit off the ceiling light as this was the closest point (behind a GFCI as are all my workshop circuits which is a bonus). The inconvenience that at least some lighting needs to be on in the workshop for the fan to run was easily offset by the ease of wiring compared to running the wire all the way to the workshop light switch. <span class="myinline">If all else fails, and if I need to run the exhaust overnight with someone sleeping in the basement next to the workshop and the 3D printer (odd as it is that the workshop light would bother that individual while the running 3D printer itself would not), I can simply temporarily unscrew the bulb.</span></div>
<div>
<br /></div>
<h4>
1. Mounting the hood above the 3D printer</h4>
<div>
My idea was to make the mount without drilling any holes in the visible part of the cabinet that doubled as the printer stand. So I used some hobby 2x2 wooden stock and steel brackets from Home Depot to improvise a stand on top of the cabinet. Here goes:</div>
<div>
<br /></div>
<div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6WnFWupocC7XEquXn214Iu-cJstmQn3stY6DYBc6CvALb7V7s95JmwEfRwcuYiKJ-Qpt2L9szgLjlwsT4_-nKGELu9T_4NrrX-FpYgkASnW4JpVyeM_aKuBGdhI7XdJzWaF7VbDrrJJA/s1600/mount.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="450" data-original-width="667" height="268" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6WnFWupocC7XEquXn214Iu-cJstmQn3stY6DYBc6CvALb7V7s95JmwEfRwcuYiKJ-Qpt2L9szgLjlwsT4_-nKGELu9T_4NrrX-FpYgkASnW4JpVyeM_aKuBGdhI7XdJzWaF7VbDrrJJA/s400/mount.PNG" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">[[[T, top surface of the IKEA cabinet (birch veneered particle board); PL, back of the IKEA cabinet (<strike>plywood</strike> thin MDF board); P, 2x2 poplar; O, 2x2 oak. Steel brackets and screws are self-explanatory.]]]</td></tr>
</tbody></table>
<br /></div>
<div>
<br />
The resulting stand wasn't 100% solid in all directions but served my purposes. (You might as well use the nearest wall stud, though.) </div>
<div>
<br /></div>
<div>
The remaining steps were to attach the microphone mount to the stand, flange then arm then gooseneck. I ended up disassembling the boom arm to be able to screw it onto the flange to avoid moving the cabinet (it is secured to the wall to avoid tipping). I fitted the hood directly to the gooseneck by drilling a carefully measured 5/8 inch hole and tightening the hood wall between two mic nuts. I may be in need of reinforcing the gooseneck in the future if it becomes limp over time.</div>
<div>
<br /></div>
<div>
<h4>
2. Mounting the fan motor </h4>
</div>
<div>
This step was rather straightforward. I noticed that the motor fits pretty tightly with the reducer on one side and the Y-connector on the other side (the latter needed some slight Dremel grinding to make a groove to accommodate a rivet head on the fan). The reducer, in turn, fit nicely into the backdraft damper, which could be connected to the inside of the duct with the aid of a 4-inch hose connector.</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div>
The outside of the duct needed to be replaced with a louvered vent which I had lying around from some previous projects. However I had no pest-proofing grille, so I ended up printing one:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCIGJtBH0cXNRwUdIJ0hskV9KaY9oB0FYYpKyeAW399HHfXTV17DBd8WH8czOfOB4GJwrVrIDB_j66ebJgGKsErBTdPxb4vgqFahma_QKAis56Uo7pQ-gsEA2bvPGBMyrZOCWwWV5s9xE/s1600/vent_mesh.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="610" data-original-width="1297" height="187" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCIGJtBH0cXNRwUdIJ0hskV9KaY9oB0FYYpKyeAW399HHfXTV17DBd8WH8czOfOB4GJwrVrIDB_j66ebJgGKsErBTdPxb4vgqFahma_QKAis56Uo7pQ-gsEA2bvPGBMyrZOCWwWV5s9xE/s400/vent_mesh.PNG" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMfAhufQRkAlHiJgAdbnEHC2lmcK6ykUhPmOprVKjDgpkfctI0X6T3k8MVUfEBCyyb8UwKchAcKKPSfgYJ3naxNArlO-9WmyX7YcQaNfe6OpoTHHrSaU3Rgg_3OepJRi5QfD6zQbc_lNQ/s1600/grille-printed.jpg" imageanchor="1" style="font-size: medium; margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="733" data-original-width="1600" height="181" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMfAhufQRkAlHiJgAdbnEHC2lmcK6ykUhPmOprVKjDgpkfctI0X6T3k8MVUfEBCyyb8UwKchAcKKPSfgYJ3naxNArlO-9WmyX7YcQaNfe6OpoTHHrSaU3Rgg_3OepJRi5QfD6zQbc_lNQ/s400/grille-printed.jpg" width="400" /></a></td></tr>
</tbody></table>
</div>
<br />
<div class="myaux">Oddly enough, some 1/3 of the grille beams had no brim around them, and seemingly started printing in thin air, but the grille ended up acceptably good. Indeed PLA can be very forgiving. Next time I'll be reprinting those in ABS or PETG (and I might need it because PLA may not be the best material to leave exposed to the elements) I'll probably use a raft.</div>
<div>
<br /></div>
<div>
An unpleasant surprise came up when it turned out that the airflow was barely enough to open the damper, limiting the exhaust functionality quite significantly. So I quickly outfitted the damper with a quick-and-very-dirty manual open/close actuator threading a piece of string through three tiny holes and tying two knots, like so:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVdjrZL5uaoewLjpGpsclz7tio6w4pYvsNh_fRUNKH9ZVTTvvDhT4ATR8OQSoW1WR91wbfHK6VOeWBPXgPXqsPB66mvY63R6X8Vg8q1QauOnUkFMQ-nNtglUHf-sJiD1LyMYLa_60gaZE/s1600/damper.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="496" data-original-width="1397" height="225" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVdjrZL5uaoewLjpGpsclz7tio6w4pYvsNh_fRUNKH9ZVTTvvDhT4ATR8OQSoW1WR91wbfHK6VOeWBPXgPXqsPB66mvY63R6X8Vg8q1QauOnUkFMQ-nNtglUHf-sJiD1LyMYLa_60gaZE/s640/damper.PNG" width="640" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
The beauty of it is that it is simple enough to do in 20 minutes, versatile enough to allow for 3 modes of operation (as seen above), is easily outfitted with visual aids, can be secured in either open or closed position using simple loop-and-nail "mechanisms", and even lends itself to automatic actuation upgrade using some kind of <a href="https://www.amazon.ca/Uxcell-a14032200ux0084-Electric-Electromagnet-Solenoid/dp/B00LBQ229Y/">solenoid electromagnet</a> <span class="myinline">(I may do such an upgrade in the future by rigging a solenoid connected to a current sensing switch from within the switch box, as I do have a few <a href="https://www.amazon.ca/gp/product/B00WS2QXG8/">current sensing transformers</a> lying around.)</span></div>
<div>
<br /></div>
<div>
<br /></div>
<h4>
3. Installing the blast gate and assembling</h4>
<div>
Probably the most invasive part was installing the blast gate through the drywall. I decided to use a shuttered gate (rather than making do with something like routing a duct directly through the wall) for several reasons:<br />
<br />
<ol>
<li>I wanted to be able to shut off the exhaust when not in use, so that I don't get extra cold air seeping onto my 3D printer (and into my basement). Also, to increase the efficiency of the workshop exhaust, I needed to be able to shut off the 3D printer branch. </li>
<li>I wanted to protect the drywall from chipping near the hole. Drywall is brittle, so anything that could wiggle inside the hole would gradually cause it to chip and crack. Not only would it damage the wall, but also the resulting dust would not add to the quality of the 3D printing or the surrounding air.</li>
<li>I wanted the connection to look as aesthetically pleasing as I could make it, so that if I decide to discontinue the use of the exhaust or the 3D printer or both, I am left with a nice looking round opening with a door that can be easily closed (rather than an ugly looking protruding piece of aluminum duct, or worse, a gaping 4-inch hole). </li>
</ol>
<br />
Cutting the hole ended up surprisingly easy with a very dull 4-inch hole saw and a handheld power drill. (Actually, the "very dull" did more good than harm. Had the saw been sharp, it would start catching in the drywall, tearing chunks off it (making a lousy looking hole), or worse, tearing the drill away from my grip and sending it flying all over the workshop. Although I might have avoided it by running the drill backwards.) I took time to carefully tape a few garbage bags with masking tape around the hole to minimize dust generation, so the hole ended up very nice. <span class="myinline">(My wife didn't welcome the idea regardless.)</span><br />
<br />
Once the hole was there, it just took some silicone caulk to glue the blast gate in place</div>
<div>
(I also reinforced the shutter stop of the gate by an M4 machine screw to prevent the door from sliding completely out if pulled too vigorously). What remained was simply attaching some 4-inch flex ducts with worm-gear clamps, sealing some joints with (insulating) duct tape, and the ductwork part was ready.</div>
<div>
<br /></div>
<h4>
4. Making and attaching the enclosure walls</h4>
<div>
To make the enclosure itself, I re-purposed a decommissioned used <a href="https://www.dollarama.com/en-CA/p-rectangular-clear-plastic-tablecloth/3018613">clear table cloth from the dollar store</a>, and made 4 curtains the following way:<br />
<br />
<ol>
<li>Cut each curtain to size Wx2H, fold to make a double-layered curtain .</li>
<li>Use an <a href="https://www.amazon.ca/gp/product/B011TS3UWQ/">impulse sealer</a> to fuse the layers of the curtain at regular intervals. <span class="myinline">(Yes this was a good excuse to buy an impulse sealer.)</span></li>
<li>Place a small weight (I used some Home Depot brass rods I had around for 4 years but never used) into the fold; seal sideways so that the weight does not slide out.</li>
<li>Attach curtains to the hood using strips of Velcro tape.</li>
<li>Put some more Velcro tape strips on the sides (to close the enclosure) as well as in the middle (to hold the curtains rolled fully or partially up, if needed). I'll have to figure out whether to make them more continuous (to make the enclosure more airtight on the sides) or more in pieces (to make the curtain easier to roll up).</li>
</ol>
</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLlKnRgkEhyrEz427bZq0DnKZw8NpCNg_WjolkRxdWUJoadgQwyDBBjtbpFiqPIyp4QGUA4VpjkGFC3b0Ii7zdrZIUnslkK6u8F8YBcrL5MRBHAqRynBBfaBCM_UHq8WvMlREhpMSVSLA/s1600/curtain.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="302" data-original-width="395" height="305" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLlKnRgkEhyrEz427bZq0DnKZw8NpCNg_WjolkRxdWUJoadgQwyDBBjtbpFiqPIyp4QGUA4VpjkGFC3b0Ii7zdrZIUnslkK6u8F8YBcrL5MRBHAqRynBBfaBCM_UHq8WvMlREhpMSVSLA/s400/curtain.PNG" width="400" /></a></div>
<br /></div>
<h3>
Final result</h3>
<div>
Well, a couple of pictures are worth a thousand words (This has been a long read anyways.)<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOkx2QbCwQooQAEdu_HoCmgBtWID7dr9kon18_LNnG-JRFTDib13SJnDksB-4YrRIZKumwLer22siexwgSiTcFaqE83LXHEn8lktKea1gE9-vZZA3L7ElbE3jyoJOHPxnWfCoBXzO_wrI/s1600/exhaust.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="573" data-original-width="1600" height="141" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOkx2QbCwQooQAEdu_HoCmgBtWID7dr9kon18_LNnG-JRFTDib13SJnDksB-4YrRIZKumwLer22siexwgSiTcFaqE83LXHEn8lktKea1gE9-vZZA3L7ElbE3jyoJOHPxnWfCoBXzO_wrI/s400/exhaust.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Workshop side (before and after). The yellow string operates the backdraft shutter.</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgH5wOd4LF2dIKTRDb3IZqrm60hjMiyFtxt7Yodl3SeUv1C8iQN1ya-Y7kvjjHGiEBvCLd0ar9zW9iBOu-Kli5hev5J89fFlf3Im5SWbzFswd1CQSAn_x57Z9k4r01UfmVmZj94-w3ilyM/s1600/final-exhaust.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="1600" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgH5wOd4LF2dIKTRDb3IZqrm60hjMiyFtxt7Yodl3SeUv1C8iQN1ya-Y7kvjjHGiEBvCLd0ar9zW9iBOu-Kli5hev5J89fFlf3Im5SWbzFswd1CQSAn_x57Z9k4r01UfmVmZj94-w3ilyM/s400/final-exhaust.jpg" width="397" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Basement side (also showing the printer in actual operation)</td></tr>
</tbody></table>
<h3>
Outlook</h3>
</div>
<div>
<br />
<div class="myaux">
<ul>
<li>The gooseneck, as well as the ABS extender sheet, are just barely adequate for the weight of the structure. The hood looks a bit flimsy and will eventually shift under use, especially when (un)rolling the curtains. I will be reinforcing the gooseneck, or maybe even foregoing it entirely, opting for another articulated arm instead. </li>
<li>The curtains design just sucks in terms of fireproofing - any flame and it's curtains for the curtains, if you excuse the pun - as well as for the printer and the basement. <span class="myinline">I may, at some point in the future, upgrade the enclosure to some flame retardant (and also more easily rollable) version. So far I am investing in a smoke detector directly above the printer. And an air quality monitor to boot.</span> </li>
<li>On the flip side, I like the transparent enclosure and easy access to the filament spool. (Most people would mount the spool outside of the enclosure anyway but I was too lazy to do so.)</li>
</ul>
</div>
</div>
Sergeihttp://www.blogger.com/profile/02237867271347372287noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-88700327192331642252020-01-11T11:40:00.003-05:002020-01-11T11:41:04.932-05:00Tamper detection flip-flop circuit and my first soldering experience<h3>
Preface</h3>
<div>
In any modest-sized family home, locks tend to accumulate - door locks, shed locks, mailbox locks, cabinet locks, closet locks, lockbox locks... Locks breed keys, keys breed frustration when they get misplaced - and they <i>always</i> get misplaced when they are used a few times a year. And even though I did decide to learn lock picking (sic!) in the meantime, to have a last-resort method to open those pesky locks, and even bought a <a href="https://www.amazon.ca/Hamkaw-Stainless-Steel-Multifunction-Locks/dp/B0816RS2R1">training set</a> (double sic!!), I never had time to practice it to the point of usability.</div>
<div>
<br /></div>
<div>
An obvious solution is a <a href="https://www.amazon.ca/gp/product/B000IO8QXG">key cabinet</a>. But, similar to the <a href="https://curiousercode.blogspot.com/2019/02/electric-cabinet-lock-and-other-small.html">jewellery drawer lock</a> I wrote about earlier, there is an inconvenience of having to lock the cabinet whenever you leave your house to cleaners or contractors and want to feel peace of mind. </div>
<div>
<br /></div>
<div>
Of course you could lock it and keep its key on your key chain. But this was not fail safe (what if your main keys get misplaced, or worse, stolen?) Instead, inspired by the <a href="https://www.litmir.me/br/?b=49323&p=21">KGB spy stories</a>, where agents would wrap a hair around their purse clasp to see if it's been opened (presumably by an opposing MI6 agent), I wanted to design an electronic detector that would alert me if the cabinet was opened in my absence.</div>
<br /><br />
<h3>
Idea</h3>
<div>
The idea that immediately comes to mind is a simple contactor, i.e. en electromagnetic relay with its coil wired in series with its normally open contacts, <a href="http://tinyurl.com/sta25oc">like so</a>:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjaPzcM7_VUU0-FUHcbEFBUK_8VaNeCfIcGRpZ-pN0MEM0jgOuZ1Fvn5VTuj2UdLrxw6u1V6XPIYJBDCIsuznlNa9D-EMjAyjI0S2Gd_vsoNUos9kM-PHbyv7VZVKKqkZZ57A4TYYq-cmo/s1600/contactor-v2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="182" data-original-width="270" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjaPzcM7_VUU0-FUHcbEFBUK_8VaNeCfIcGRpZ-pN0MEM0jgOuZ1Fvn5VTuj2UdLrxw6u1V6XPIYJBDCIsuznlNa9D-EMjAyjI0S2Gd_vsoNUos9kM-PHbyv7VZVKKqkZZ57A4TYYq-cmo/s1600/contactor-v2.png" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div class="myaux">
The operation is pretty simple: the relay K1 is initially de-energized because its contacts are normally open; as soon as the (discreetly located) Start button is pushed, contacts are bypassed, the relay is energized and maintains the contacts closed. At that point the circuit is "armed" (signaled by the LED illuminating as shown above; R1 may be needed to limit the LED current). This goes on indefinitely until the circuit is broken (say by a <a href="https://en.wikipedia.org/wiki/Reed_switch">reed switch</a> attached to the door of the key cabinet, and I had a <a href="https://www.amazon.ca/gp/product/B01M8JCGSO/">few of them lying around</a>). The LED goes out, signalling that the cabinet was opened. The diode D1 is the <a href="https://en.wikipedia.org/wiki/Flyback_diode">flyback diode</a>, especially important to protect the reed switch from arcing damage.</div>
<div>
<br /></div>
<div>
The drawbacks of this set-up are too obvious. The relay is an overkill to power a single LED (the relay coil consumes about 10 times more power). In addition, there is no way of telling if the LED went out because someone opened the cabinet or because the battery went out (and powering it from a wall adapter is a non-starter because it would give a false alarm for every tiniest power outage.) </div>
<div>
<br /></div>
<div>
To solve the first drawback, I remembered I had an <a href="https://en.wikipedia.org/wiki/Opto-isolator">optocoupler</a> lying around (the white 6-pin chip from <a href="https://www.amazon.ca/gp/product/B01IGGP7Z2/">this kit</a>), which can work like a relay but for a fraction of power. To solve the second drawback I remembered that I had also ordered <a href="https://www.amazon.ca/gp/product/B00TNJ3NH6/">some MOSFETs</a> for an earlier <a href="https://curiousercode.blogspot.com/2017/06/the-saga-of-old-sears-craftsman-garage.html">garage door opener mod project</a> and ended up using bipolar transistors instead. So the resulting circuit was something <a href="http://tinyurl.com/y6gcl2qm">like this (the link opens in a live circuit simulator)</a> :</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGB1ngJfVuoDcn6YLiIhSG4SJPx9TFwrQ0N4CTF6u1mrpRrPVht4THLs83vbA0kI3-sFrGlKGL90TUgNsRp2tu15mcHKSs71BegQnYOHn50NTiu9QlUurPgf94u8ZZF7yG724DeWni7lo/s1600/tamper-v1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="318" data-original-width="380" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGB1ngJfVuoDcn6YLiIhSG4SJPx9TFwrQ0N4CTF6u1mrpRrPVht4THLs83vbA0kI3-sFrGlKGL90TUgNsRp2tu15mcHKSs71BegQnYOHn50NTiu9QlUurPgf94u8ZZF7yG724DeWni7lo/s1600/tamper-v1.png" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div class="myaux">
Again the operating principle is pretty simple. The optocopupler Q1 takes on the role of the relay K1; the resistor R1 in the previous diagram is split between R1 and R2 here, creating a voltage divider is such a way that when current flows through Q1, the gate of the MOSFET Q2 has just the right potential to close Q2. When Q1 is de-energized, Q2 becomes opened and illuminates the alarm LED through the resistor R3. </div>
<br /><br />
<h3>
Implementation and breadboard</h3>
<div>
When I needed to spend a few hours at a car dealership waiting for my car to be serviced, I took along a few parts, a multimeter and a breadboard and tried to implement my circuit (funny as it looked in the dealership lounge). After a few trials I ended up with this (<a href="http://tinyurl.com/qmhdsr9">again please feel free to play in the simulator</a>):</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgovks-PPD7OmpitXbUg8i5LcHIbpbXLAnN2EiPDcCUr_haMH5gNmeiJWodGdFY7W1VjwbObBD4NjsYnmcCiCzfbQngD-IF36RFSMO5k8b_yDo-rEW9km09B96vMW71XP6_J7Q41S439Do/s1600/tamper-v2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="311" data-original-width="381" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgovks-PPD7OmpitXbUg8i5LcHIbpbXLAnN2EiPDcCUr_haMH5gNmeiJWodGdFY7W1VjwbObBD4NjsYnmcCiCzfbQngD-IF36RFSMO5k8b_yDo-rEW9km09B96vMW71XP6_J7Q41S439Do/s1600/tamper-v2.png" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div class="myaux">
The only modification, aside from tuning the values of the resistors to suit my actual optocoupler and MOSFET, came from the fact that my optocoupler <a href="http://www.vishay.com/docs/81181/4n35.pdf">had a separate base pin</a> for the phototransistor, and leaving it floating meant that it could get energized at random upon power-up (defeating all the purpose of the circuit). So I had to introduce the D1-R4 circuit to pull the base pin down just so that Q1 always started with closed phototransistor (in the simulator, it looks detached because the optocoupler there is a generic part having no base lead). </div>
<div>
<br /></div>
<div>
In the breadboard, the circuit looked like this (the red LED is actually D1 - I did not think I would need an ordinary diode, so I did not bring one to the shop and had to use a LED instead.):</div>
<div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhW8NvA3DrfjCm2j8BctIX9yRp7R1wXyKe72VxOBGuu7pySsOclVPFk21YdkzRDoD9XttDfWmrX9U_hnZ0DB2Vck1WY-qe1bJOMCcbFtjTxpbV05100a3aNKUl7-XEsEEH9GcFVmvZrWx4/s1600/20190613_170133.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1017" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhW8NvA3DrfjCm2j8BctIX9yRp7R1wXyKe72VxOBGuu7pySsOclVPFk21YdkzRDoD9XttDfWmrX9U_hnZ0DB2Vck1WY-qe1bJOMCcbFtjTxpbV05100a3aNKUl7-XEsEEH9GcFVmvZrWx4/s320/20190613_170133.jpg" width="203" /></a></div>
<br /></div>
<div>
<br /></div>
<br /><br />
<h3>
Soldering and final installation</h3>
<div>
Some 6 months later (yes, because small kids);I found a bit of more time to attempt to actually solder it together on a <a href="https://www.amazon.ca/gp/product/B00NQ37V0K/">protoboard PCB</a>. I had never soldered anything serious before, so I made a few bad mistakes along the way:</div>
<div>
<ol>
<li>I learned that you cannot connect two adjacent pads with just a drop of solder - needed to use a tiny bit of wire, or strip a component lead leaving 1-2 mm slack and bending it to extend to an adjacent pad.</li>
<li>I learned that it is not possible to melt more than one hole simultaneously unless you had a very specialized tip and super stable hands. So pre-filling the holes for the MOSFET (or worse, the optocoupler) was a disaster and I deeded to clean out the holes again using a desoldering pump (good I had a very basic one included with the soldering kit); a few times even this wasn't working and I had to resort to a Dremel with a very tiny drill bit. All in all, a <a href="https://www.amazon.ca/Stanz-Basic-Helping-Third-Magnifier/dp/B016C8RWQI/">third hand station</a> I bought a few years before proved very useful here (finally).</li>
<li>My decision to place components to both sides of the PCB (LEDs and the button on one side, the rest on the other) was a mistake. It would have been cleaner and more compact with everything on the same side. </li>
</ol>
Surprisingly, it was <i>much </i>easier for me to get a device working on the breadboard than in the soldered form. It proved quite hard to ensure consistently good soldering quality. I spent a few oddball hours during two days tracing and correcting bad solder points, going through some moments of despair and almost wanting to get back to the breadboard. </div>
<div>
<br /></div>
<div>
But patience and perseverance won (and, oddly but fortunately, the components endured my multiple attempts). Here goes (note that D1 is now a proper diode :) :</div>
<div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8GrrTI8FjWGmtYIdimN9eO5aQz31AAP6gdkZLBrcUzBrNwawTjjagP1uXgmE0l_Ec87djkduN0cTIu2gWTiF9JeROaXQRJZRee5YcDbxsbujqux73kr0-kPTQoULon1MeWASICvbFcfo/s1600/solder.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1112" data-original-width="1600" height="222" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8GrrTI8FjWGmtYIdimN9eO5aQz31AAP6gdkZLBrcUzBrNwawTjjagP1uXgmE0l_Ec87djkduN0cTIu2gWTiF9JeROaXQRJZRee5YcDbxsbujqux73kr0-kPTQoULon1MeWASICvbFcfo/s320/solder.jpg" width="320" /></a></div>
<br /></div>
<div>
<br /></div>
<div>
The remaining step was to install it in some kind of case. I did not have anything fancy like a 3D printer so I used a <a href="https://www.amazon.ca/gp/product/B014GN10CS/">pre-bought plastic box</a> to put the board inside using good old-fashioned M3 screws. Unfortunately, the batteries did not fit on the inside (they would if I had not made the mistake #3 above). So outside they went, on the back of the box. The final result looks like this:</div>
<div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1cNEF-y8Yusz0Xxo4LRjyg2nYfQ03qISy9Sg03F5xHMyeHNH0XD2Sj6gPejS6yekAWq-E0C_NwQDgJL_DODS1kmLNE2t8tAuqyf24qmPu-xWIt4XebIGkC6mbKCW1rBzOED2SqQJjCCg/s1600/final.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="900" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1cNEF-y8Yusz0Xxo4LRjyg2nYfQ03qISy9Sg03F5xHMyeHNH0XD2Sj6gPejS6yekAWq-E0C_NwQDgJL_DODS1kmLNE2t8tAuqyf24qmPu-xWIt4XebIGkC6mbKCW1rBzOED2SqQJjCCg/s320/final.jpg" width="180" /></a></div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div class="myaux">
The Start button is hidden behind the fourth screw (the one beside the LEDs). It is shorter than the remaining three screws that hold the board in place. To arm, the screw needs to be removed, and the button can then be pressed through its hole.</div>
<div>
<br /></div>
<div>
In operation, it looks like this:</div>
<div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen='allowfullscreen' webkitallowfullscreen='webkitallowfullscreen' mozallowfullscreen='mozallowfullscreen' width='320' height='266' src='https://www.blogger.com/video.g?token=AD6v5dz16HVwynZmWJJ-tnfaJ8wX-7JtfBRdqfGQeURYoDXepANORoQFjJlolvT8gBXsadpkh66YyTaDQu-O2zrSsw' class='b-hbp-video b-uploaded' frameborder='0'></iframe></div>
<br /></div>
<div>
<br /></div>
<div class="myaux">
Bonus: The current consumption of the circuit is around 2 mA (two milliamps), in either state, and most of it is the LEDs. Which means that four AAA batteries would power it through months on end; if this proves not enough, I'll put four D batteries and will probably only need to replace them once every few years.</div>
<div>
<br /></div>Sergeihttp://www.blogger.com/profile/02237867271347372287noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-23079939129333829352019-12-13T14:29:00.003-05:002019-12-13T15:03:37.668-05:00Heartbeat/watchdog service in MS ExcelWe have already covered automating Excel tasks at work -- and indeed it can be a life saver (or rather, work-life balance saver) <i>not</i> to have to be present at work <i>every single day at time XX:XX no matter what</i> only to have to push a few buttons.<br />
<br />
However, there is an inconvenience that no automation is 100% fool-proof, especially in a corporate environment. The automation computer may be restarted (by your colleagues or your IT department who decided to push some security updates). It may lose power (because the cleaning personnel got a bit too vigorous with their brooms). It may crash, or freeze, or have a hardware failure. It is particularly nasty is the automation computer is a shared terminal that only gets used once in a while, so a user log-off due to restart may not be immediately detected. And of course once the user is logged off, no code nothing written by that user can be invoked any more, and needless to say that admin access to all computers in the corporate is strictly verboten.<br />
<br />
One way of getting around the inconvenience would be to implement a "<a href="https://en.wikipedia.org/wiki/Heartbeat_(computing)">heartbeat</a>" or "<a href="https://en.wikipedia.org/wiki/Watchdog_timer">watchdog</a>" service that would periodically query the availability of your automation machine and "phone home" if it goes offline.<br />
<br />
The general idea is as follows:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibYOrhS6vZFtGkhO6W0Glosp_xzclt-NKipivbwkO36v913Ta4s6deoWG9pO2j8c2KQ3nifYlen62pyzQo8xbziBqffqu5iPWDHgm_s6Rcwa745pDXxWh_DMS_AuydJORoyCllzPwblTY/s1600/heartbeat.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="473" data-original-width="757" height="248" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibYOrhS6vZFtGkhO6W0Glosp_xzclt-NKipivbwkO36v913Ta4s6deoWG9pO2j8c2KQ3nifYlen62pyzQo8xbziBqffqu5iPWDHgm_s6Rcwa745pDXxWh_DMS_AuydJORoyCllzPwblTY/s400/heartbeat.PNG" width="400" /></a></div>
<ul>
<li>The "server" is an Excel workbook that does nothing and resides in the user's <i>private</i> folder on the <i>shared</i> drive (most corporate environments have that), so that it's accessible from any computer in the office but only by the particular user. This worksheet contains no code, but is configured to open at startup on the automation machine.</li>
<li>The "client" is another Excel workbook that runs on the user's regular desktop (which is assumed to be operational). </li>
<li>The client polls the server by trying to open the server workbook (assuming that if it's locked for editing, it's open on the automation machine so the latter is operational). </li>
<li>Should the server workbook become unlocked, this means that the automation machine has logged the user off and automation won't run. The client then informs the user by sending an email. </li>
</ul>
To determine whether a file is open on the automation machine, I use the following macro (adapted from <a href="https://www.mrexcel.com/board/threads/vba-if-file-is-open.923421/">this idea</a>):<br />
<br />
<br />
<pre class="brush:vb">Function IsFileOpen(filename As String)
Dim filenum As Integer, errnum As Integer
On Error Resume Next ' Turn error checking off.
filenum = FreeFile() ' Get a free file number.
' Attempt to open the file and lock it.
Open filename For Input Lock Read As #filenum
Close filenum ' Close the file.
errnum = Err ' Save the error number that occurred.
On Error GoTo 0 ' Turn error checking back on.
' Check to see which error occurred. Open for
IsFileOpen = (errnum = 70)
End Function
</pre>
<br />
The periodic polling code may look something like this:<br />
<br />
<br />
<br />
<pre class="brush:vb">Sub Heartbeat()
Dim monitor As String
Dim span, tick, fail, restart As Integer
With ThisWorkbook.Worksheets("Control")
.Calculate
monitor = .Range("monitor").Value
span = .Range("span").Value: fail = .Range("numfail").Value: restart = .Range("autoreset").Value
If IsFileOpen(monitor) Then
.Range("tick").Value = 0
.Range("retick").Value = 0
Call LogMessage("INFO", "Heartbeat server detected")
Else
.Range("tick").Value = .Range("tick").Value + 1
Call LogMessage("WARN", "Heartbeat server NOT detected")
End If
If .Range("tick").Value < fail Then
Application.OnTime Now + span / 86400, "Heartbeat"
Else
Call LogMessage("FAIL", "Heartbeat server has failed, REPORTING FAILURE")
Call SendEmail(.Range("ReportMail"))
Application.OnTime Now + restart / 86400, "RestartHeartbeat"
End If
End With
End Sub
</pre>
<br />
The idea is that the macro will restart itself using <span class="mycode">Application.OnTime</span> if all goes well, until the server is not detected several times in a row (to guard against false positives due to intermittent network failures). In that case, a report is set via Outlook (adapted from <a href="http://www.excelmacroclasses.com/author/excelmacroclasses/page/27/">here</a> and possibly <a href="http://www.rondebruin.nl/">here</a>) using this function:<br />
<br />
<br />
<pre class="brush:vb">Sub SendEmail(params As Range)
Dim OutApp As Object, OutMail As Object, strbody As String, toaddr As String, ccaddr As String, subj As String
strbody = "": toaddr = "": ccaddr = "": subj = "":
Dim here As Range
Set OutApp = CreateObject("Outlook.Application")
Set OutMail = OutApp.CreateItem(0)
Set here = params(1, 1)
While Len(here.Value) > 0
If UCase(Left(Trim(here.Value), 2)) = "TO" Then toaddr = here.Offset(0, 1).Value
If UCase(Left(Trim(here.Value), 2)) = "CC" Then ccaddr = here.Offset(0, 1).Value
If UCase(Left(Trim(here.Value), 2)) = "SU" Then subj = here.Offset(0, 1).Value
If UCase(Left(Trim(here.Value), 2)) = "BO" Then strbody = here.Offset(0, 1).Value
If UCase(Left(Trim(here.Value), 1)) = ":" Then strbody = strbody & vbNewLine & here.Offset(0, 1).Value
Set here = here.Offset(1, 0)
Wend
On Error Resume Next
With OutMail
.to = toaddr
.CC = ccaddr
.BCC = ""
.Subject = subj
.Body = strbody
.Send 'or use .Display
End With
On Error GoTo 0
Set OutMail = Nothing: Set OutApp = Nothing
End Sub
</pre>
<br />
Note that the parameters for the email are conveniently stored on the spreadsheet for easy editing, so that only the range pointing to the parameter block needs to be passed. The email can then be easily defined using something like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicBPXcw2fTAK5Y8Cg8D3XgHzkeb0B5_yIFHAmT0CgQudPbfCcEX2xxZ6lXvO8uBcU2S73akxH8uvDG02MIlcnQv-Dp488ChTo-lCgg7yK4E3_DqeJ-YC1IabOAkx2xQccwhX9CLhBT0rA/s1600/excel-report.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="169" data-original-width="419" height="129" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicBPXcw2fTAK5Y8Cg8D3XgHzkeb0B5_yIFHAmT0CgQudPbfCcEX2xxZ6lXvO8uBcU2S73akxH8uvDG02MIlcnQv-Dp488ChTo-lCgg7yK4E3_DqeJ-YC1IabOAkx2xQccwhX9CLhBT0rA/s320/excel-report.PNG" width="320" /></a></div>
<br />
After the failure report is sent, a slightly different macro is invoked:<br />
<br />
<br />
<pre class="brush:vb">Sub ReStartHeartbeat()
Dim monitor As String, span As Integer, autoreset As Integer
With ThisWorkbook.Worksheets("Control")
.Calculate
monitor = .Range("monitor").Value
span = .Range("span").Value
autoreset = .Range("autoreset").Value
If IsFileOpen(monitor) Then
.Range("tick").Value = 0
.Range("retick").Value = 0
Call LogMessage("OK", "Heartbeat server restarted, monitoring resumed")
Call SendEmail(.Range("ReSetupMail"))
Application.OnTime Now + span / 86400, "Heartbeat"
Else
.Range("retick").Value = .Range("retick").Value + 1
Call LogMessage("FAIL", "Heartbeat server NOT restarted, keeping on trying")
Call SendEmail(.Range("ReSetupErrorMail"))
Application.OnTime Now + autoreset / 86400, "ReStartHeartbeat"
End If
End With
End Sub
</pre>
<br />
Here the idea is to keep reminding the user that the server is still down, and getting back on track once the user has successfully corrected the problem.<br />
<br />
The entire process is set into motion by two macros:<br />
<br />
<br />
<pre class="brush:vb">Sub StartHeartbeat()
Dim monitor As String, span As Integer
Call LogMessage("INFO", "Service started")
With ThisWorkbook.Worksheets("Control")
.Calculate
monitor = .Range("monitor").Value
span = .Range("span").Value
If IsFileOpen(monitor) Then
.Range("tick").Value = 0
Call LogMessage("OK", "Heartbeat server detected, monitoring started")
Call SendEmail(.Range("SetupMail"))
Application.OnTime Now + span / 86400, "Heartbeat"
Else
Call LogMessage("WARN", "Service not started, heartbeat server not detected")
Call SendEmail(.Range("SetupErrorMail"))
End If
End With
End Sub
</pre>
<br />
<pre class="brush:vb">Private Sub Workbook_Open() ' Put this in the Workbook code rather than Module1
If ThisWorkbook.ReadOnly Then Exit Sub ' DO NOT execute if read only OR
If Workbooks.Count > 1 Then
Call LogMessage("WARN", "Started in non-clean session, entering setup mode. Service NOT starting. For production, run in a CLEAN session.")
ThisWorkbook.Save
Exit Sub ' ONLY execute in clean, dedicated session
End If
ThisWorkbook.Worksheets("Control").Calculate
Application.OnTime Now + TimeValue("00:00:05"), "StartHeartbeat"
End Sub
</pre>
The safeguards in place make sure that opening the client for debugging or viewing (or by accident) do not start spurious monitoring processes (<span class="mycode">.OnTime</span> is nasty, once set it will persist untill that particular Excel session is ended, <i>even after</i> the workbook containing the <span class="mycode">.OnTime</span> was closed). So the client only starts if the file is only opened in a clean, dedicated Excel session.<br />
<br />
<div class="myaux">
Finally, an auxiliary subroutine, purely aesthetic, is used to log the monitoring actions. Here goes:<br />
<br />
<br />
<pre class="brush:vb">Sub LogMessage(code As String, msg As String)
Dim here As Range, c As Integer, level As Integer
level = 255
If UCase(Left(Trim(code), 3)) = "INF" Then level = 10
If UCase(Left(Trim(code), 2)) = "OK" Then level = 5
If UCase(Left(Trim(code), 3)) = "WAR" Then level = 2
If UCase(Left(Trim(code), 3)) = "FAI" Then level = 1
If UCase(Left(Trim(code), 3)) = "ERR" Then level = 0
For c = 1 To 2
Set here = ThisWorkbook.Worksheets(IIf(c = 1, "Log", "Errors")).Range("A1")
If c = 2 And level >= 2 Then Exit For
While Len(here.Value) > 0
Set here = here.Offset(1, 0)
Wend
With ThisWorkbook.Worksheets("Control")
.Calculate
here.Value = IIf(Len(code) > 0, code, "___")
here.Offset(0, 1).Value = Now
here.Offset(0, 2).Value = .Range("monitor").Value
here.Offset(0, 3).Value = .Range("span").Value
here.Offset(0, 4).Value = .Range("numfail").Value
here.Offset(0, 5).Value = .Range("tick").Value
here.Offset(0, 6).Value = .Range("autoreset").Value
here.Offset(0, 7).Value = .Range("retick").Value
here.Offset(0, 8).Value = msg
Set here = ThisWorkbook.Worksheets(IIf(c = 1, "Log", "Errors")).Range(here, here.Offset(0, 8))
Select Case level
Case 10
here.Font.Color = RGB(200, 200, 200)
Case 5
here.Font.Color = RGB(0, 128, 0)
Case 2
here.Font.Color = RGB(255, 128, 0)
Case 1
here.Font.Color = RGB(128, 0, 0)
Case 0
here.Font.Color = RGB(255, 0, 0)
Case Else
here.Font.Color = RGB(255, 0, 255)
End Select
End With
If c = 2 Then ThisWorkbook.Save ' only save on error
Next c
End Sub
</pre>
<br /></div>
Sergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-73385749583904370252019-11-08T12:40:00.001-05:002019-11-08T17:41:27.481-05:00House hunting in the cloudLet us continue the tradition of following a <a href="https://curiousercode.blogspot.com/2019/09/esoteric-multiplication-and-almost.html">low-level language coding post</a> with <a href="https://curiousercode.blogspot.com/search/label/wolfram%20language">really high-level</a>. This time, with a cloud to boot.<br />
<br />
Anyone who was apartment and house hunting knows that it is time consuming and hard to go through individual listings. Extracting trends from a large number of them can be very time-consuming and involving a lot of manual input.<br />
<br />
It would be really helpful to have a took that would extract the features you need from a listing feed, and visualize it in your preferred way, like so:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5vw-njrGRgrDwtHbVrj8PCRbAyLpL_O9sx5E8o9kbeXfPmR1uAumrMousYXrNiRbfKrmVRspCSCWf7P3Ro-kSy407UVHZgR3Y7AXrjOFpjNUP7U7OceyAjmcQ9kTCVASR3kwZFQZtcF0/s1600/title.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="180" data-original-width="362" height="159" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5vw-njrGRgrDwtHbVrj8PCRbAyLpL_O9sx5E8o9kbeXfPmR1uAumrMousYXrNiRbfKrmVRspCSCWf7P3Ro-kSy407UVHZgR3Y7AXrjOFpjNUP7U7OceyAjmcQ9kTCVASR3kwZFQZtcF0/s320/title.PNG" width="320" /></a></div>
<div style="text-align: center;">
<br /></div>
<br />
<br />
We can actually do this with a little bit of coding entirely in <a href="https://www.wolframcloud.com/" target="_blank">Wolfram Cloud</a>. Let's get started with a sample Toronto <a href="https://www.getwhatyouwant.ca/how-to-read-a-toronto-mls-condo-listing">MLS listing</a> selection (the kind you will be receiving from a real estate agent), looking like so (sorry for a tall image).<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgo1mygxo2bKmct8lCLfz6nNYrsfu-0yZoMGe2Opwyzb9f-almEx6rD4W0kJ_GZeILbp3AyJxULWPRBe-ZvlwajtuiA_EexGijiwHN_u4STj_-takEXwFutVBSsdaHeqYpwdAuyfIQf3zM/s1600/MLSTall.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="744" data-original-width="280" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgo1mygxo2bKmct8lCLfz6nNYrsfu-0yZoMGe2Opwyzb9f-almEx6rD4W0kJ_GZeILbp3AyJxULWPRBe-ZvlwajtuiA_EexGijiwHN_u4STj_-takEXwFutVBSsdaHeqYpwdAuyfIQf3zM/s320/MLSTall.PNG" width="120" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="text-align: center;">
<br /></div>
<br />
So the task is (1) to web-scrape the listing web page for information, and (2) to visualize it in human digestible form.<br />
<br />
Using <a href="https://blog.wolfram.com/2018/03/02/web-scraping-with-the-wolfram-language-part-1-importing-and-interpreting/">this nice Wolfram Language scraping guide</a>, we can get started by simply grabbing the table data in one line, like so:<br />
<br /><br />
<pre class="brush:math">
url = "http://v3.torontomls.net/Live/Pages/Public/Link.aspx?Key=...&App=TREB";
structured = Import[url,"Data"];
mlstable = structured[[1,2]];
addresses = Transpose[mlstable][[2]];
rawprices = Transpose[mlstable][[4]];
prices = ToExpression[StringReplace[#,{"$"->"",","->""}]]&/@rawprices;
</pre>
<br /><br />
Now we have the list of street addresses and prices. To geo-locate the addresses, the easiest way is to complete each address with city and country info, ans then use <span class="mycoode">Interpreter</span>:<br />
<br /><br />
<pre class="brush:math">
locations = Map[Interpreter["StreetAddress"][#<>", Mississauga ON Canada"]&, addresses];
</pre>
<br /><br />
We can then convert the list of prices into a list of colored pins where color is determined by the house price, using a slightly modified example from <a href="https://reference.wolframcloud.com/language/ref/GeoMarker.html"><span class="mycode">GeoMarker</span> documentation</a>:<br />
<br /><br />
<pre class="brush:math">
pin[color_]:= Graphics[GraphicsGroup[{FaceForm[color],EdgeForm[Black],
FilledCurve[{{Line[Join[{{0, 0}}, ({Cos[#1], 3 + Sin[#1]} &) /@
Range[-((2 Pi)/20), Pi + (2 Pi)/20, Pi/20], {{0, 0}}]]}, {Line[(0.5 {Cos[#1], 6 + Sin[#1]} &) /@
Range[0, 2 Pi, Pi/20]]}}]}]]
rcfun[sc_] :=Blend[{{0,Green},{0.5,RGBColor[0.75,0.75,0]},{1,Red}},sc]
pins = Map[pin[rcfun[(#-900000)/(1300000-900000)]]&,prices];
</pre>
<br /><br />
It only remains to convert the list of geographical coordinates and pins to <span class="mycode">GeoMarker</span> and filter out failed address lookups (as well as missed lookups, defined as those more than say 10 miles away from the arbitrarily chosen city center), like so<br />
<br /><br />
<pre class="brush:math">
markertable = MapThread[GeoMarker[#1,#2]&,{locations,pins}];
home = Interpreter["StreetAddress"]["Square One, Mississauga ON Canada"];
goodmarkers = Select[markertable,(!FailureQ[#[[1]]] && GeoDistance[#[[1]],home][[1]]<10)&];
</pre>
<br /><br />
In my example, 94 markers out of 99 remain as "good". Then, we simply plot the markers on a map using <span class="mycode">GeoGraphics</span>:<br />
<br /><br />
<pre class="brush:math">
gp = GeoGraphics[{"Mississauga",Append[goodmarkers,GeoMarker[home,pin[White]]]}];
gins=DensityPlot[(x*1000-900000)/(1300000-900000),{y,0,1},{x,900,1300},ColorFunction->rcfun,AspectRatio->5,FrameTicks->{None,Automatic},Background->RGBColor[1,1,1,0.5],FrameStyle->Directive[Thick],LabelStyle->Normal];
Show[gp,Epilog->Inset[gins,Scaled[{1,0}],Scaled[{1,0}],0.028]]
</pre>
<br /><br />
<br /><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj94HrlepTvv1ijpWxgMMBjKX7yAvLLRvCJubVEX489GB_QT0M7r1UEWwqImm9g1WQziFBMpuGjRPwvtIur7-gbiEvbclqi8uOgpdDN9AUlm99d_pg4642PBcxe8af05svLkk2goj_ei70/s1600/map-price.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="328" data-original-width="420" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj94HrlepTvv1ijpWxgMMBjKX7yAvLLRvCJubVEX489GB_QT0M7r1UEWwqImm9g1WQziFBMpuGjRPwvtIur7-gbiEvbclqi8uOgpdDN9AUlm99d_pg4642PBcxe8af05svLkk2goj_ei70/s1600/map-price.png" /></a></div>
<div style="text-align: center;">
<br /></div>
<br />
In the second example let us color code the markers using price per square foot. Note that the square footage of the houses is not in the table, so we need to parse individual listings. So we need to do a more complicated web scraping:<br />
<br /><br />
<pre class="brush:math">
url = "real_estate.html"; xml = Import[url,"XMLObject"];
formitems=Cases[xml,XMLElement["span",{"class"->"formitem formfield"},x_]->x,Infinity];
sqfeet=ToExpression[Last[StringSplit[#,"-"]]]&/@((If[#[[3]]=={},"0-0",#[[3,1]]])&/@Extract[formitems,(#+{0,1})&/@Position[formitems,XMLElement["label",{},{"Apx Sqft:"}]]] )
</pre>
<br /><br />
<div class="myaux">
Note several things about this code:
<ul>
<li>The specific criteria to supply to <span class="mycode">Cases</span> have been determined by inspecting the page code in the browser; in this case all the information bearing fields conveniently are <span class="mycode"><span></span> tags with classes <span class="mycode">formitem formfield</span>. Your particular case will be different.</li>
<li>In the last line, <span class="mycode">Extract</span> basically retrieves "every element following a label saying <span class="mycode">Apx Sqft:</span>". Again your case will be different, and I admit that this is not the only way to get to the right info.</li>
<li>Local HTML file is used instead of a live URL. This is a trick dome to work around asynchronous deferred loading of listings on Toronto MSL website; if live URL were used, the scraper would only retrieve some 25 listings. The HTML file is obtaied by loading the MLS link, scrolling all the way down (not too fat so that all listings have a chance to load), then saving the webpage as a complete package and loading its HTML file into Wolfram Cloud (or creating a text file in the cloud and copy-pasting, or hosting it locally and making it accessible to Wolfram Cloud)</li>
<li>The last portion, involving <span class="mycode">Last[StringSplit[...]]</span> is needed to convert approximate designations like "1500-2000" into a number 2000.</li>
</ul>
</div><br />
After this, the code is familiar, except that marker filtering should now include filtering out the listings without square feet information:<br />
<br />
<br /><br />
<pre class="brush:math">
pricesperfoot = Quiet[prices/cfeet];
score=Quiet[(pricesperfoot-250)/(800-250)];
scorepins=Map[pin[rcfun[#]]&,score];
smarkertable = MapThread[GeoMarker[#1,#2]&,{locations,scorepins}];
sgoodmarkers=Select[smarkertable,(!FailureQ[#[[1]]] && GeoDistance[#[[1]],home][[1]]<10
&& NumberQ[#[[2,1,1,1,-1,-1]]])&];
gs=GeoGraphics[{"Mississauga",Append[sgoodmarkers,GeoMarker[home,pin[White]]]}];
Show[gs,Epilog->Inset[ginss,Scaled[{1,0}],Scaled[{1,0}],0.028]]
</pre>
<br /><br />
Here's the final result:
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg3S_0aE3deQwd9eg-GGpOoTX4aPjGom8sLrpCr2WHUXnKlX8Wb_4nXXMnecwMls538rYbyM8q5Dfi55sqhXgmm94w3lkuP4YEkE8HvZ9noVFcFPWUZvyB-YVQu3T7Rjelx_hv-vg0tQsU/s1600/map-sqft.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="326" data-original-width="420" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg3S_0aE3deQwd9eg-GGpOoTX4aPjGom8sLrpCr2WHUXnKlX8Wb_4nXXMnecwMls538rYbyM8q5Dfi55sqhXgmm94w3lkuP4YEkE8HvZ9noVFcFPWUZvyB-YVQu3T7Rjelx_hv-vg0tQsU/s1600/map-sqft.png" /></a></div>
<br />
<br />
<br />
<div class="myaux">This is only an example I spent about an hour coding, another 1-2 polishing and another 1-2 hours writing about. Your mileage may vary. By the same token you can easily visualize houses according to any score you compute (such as "price per score determined by adding the number of bedrooms and half the number of washrooms"). You can also add multidimensional visualization, where a pin's size, or border color, or shape, or all of these, would convey different information. You can use geo-location and scrape some other website to score neighborhoods and show the "best bang for the buck" according to that score. You can build a linear regression machine-learning house pricing service. If you are dragged into the boredom of house hunting, there are always ways to make some colorful fun out of it :) .</div>Sergeihttp://www.blogger.com/profile/02237867271347372287noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-72041976367494109032019-09-26T12:02:00.003-04:002019-09-26T16:45:59.588-04:00Esoteric multiplication and an (almost) failed exerciseIn the multiverse of programming languages it's hard not to mention <a href="https://en.wikipedia.org/wiki/Brainfuck">Brainf*ck</a>, purposely written to have a minimal set of commands needed to (theoretically) program just about any problem out there thanks to its <a href="https://en.wikipedia.org/wiki/Turing_completeness">Turing completeness</a>. The set is actually so minimal that I can include all of it, below. For non-programmers it would be useful to imagine an infinite linear magnetic tape with a read/write head able to move along it, like so:<br />
<br />
<pre class="myaux mycode">Command C-equivalent Action
> ++ptr; Move the head one position to the right
< --ptr; Move the head one position to the left
+ ++*ptr; Increment the byte of data at the head position
- --*ptr; Decrement the byte of data at the head position
. putchar(*ptr); Write a keyed-in byte of data at the head position
, *ptr=getchar(); Read and display the byte of data at the head position
[ while(*ptr){ If the byte at the head position is zero,
rather than executing the next command,
jump forward to the matching ]
] } If the byte at the head position is non-zero,
rather than executing the next command,
jump back to the matching [
</pre>
<br />
Well I mentioned it in conversations so often that it would have been a shame not to have tried it hands on. I took on a randomly chosen problem, namely writing a program that would multiply two single-digit numbers, totally out of the zone, somewhere in between spending last half an hour of the working day, thinking during my commute and in between playing and walking with the kid(s), and some half an hour debugging after the said kids went to sleep.<br />
<br />
I got inspired by the addition code snippet, namely, <span class="mycode">[->+<]</span>, and tried to generalize it to summing several single digits in a row. My first example, <span class="mycode">[[->+<]>]</span>, expectedly resulted in an infinite loop; however a slightly more complicated <span class="mycode">[<[->+<]>>]</span> ended up working fine.<br />
<br />
My next challenge was how to replicate one of the operands in <span class="mycode">m*n</span> an arbitrary number of times, i.e. to translate the memory layout <span class="mycode">[m n]</span> into something like <span class="mycode">[m m ... m]</span> where <span class="mycode">m</span> would be repeated <span class="mycode">n</span> times. After a few unsuccessful tries a working algorithm was:<br />
<br />
<ol><li>Replicate one operand 1 time, then 2 times, then 3 times, ... all the way to <span class="mycode">m_max</span> times. This is somewhat ugly but realizable using something like<br />
<div class="mycode">
[- >>>>>>> >+>>>>>> >+>+>>>>> >+>+>+>>>> >+>+>+>+>>> >+>+>+>+>+>><br />
<<<<<<< <<<<<<< <<<<<<< <<<<<<< <<<<<<< <<<<<<<]</div>
<br /></li>
<li>Replicate the second operand, decreasing it on every iteration, so that for <span class="mycode">n</span>, zero is reached in front of <span class="mycode">n</span> copies of <span class="mycode">m</span>, like so<br />
<span class="mycode">[[- >>>>>>> + <<<<<<<] >>>>>>>-]</span><br />
<br /></li>
<li>Apply our previously written "sum all numbers until zero" <span class="mycode">[<[->+<]>>]</span><br />
<br /></li></ol>
Taken together and surrounded with some I/O, the final program ended up being<br />
<br />
<div class="mycode">
,------------------------------------------------> read and convert n<br />
,------------------------------------------------> read and convert m<br />
<[- >>>>>>> >+>>>>>> >+>+>>>>> >+>+>+>>>> >+>+>+>+>>> >+>+>+>+>+>><br />
<<<<<<< <<<<<<< <<<<<<< <<<<<<< <<<<<<< <<<<<<<] replicate m<br />
<[[- >>>>>>> + <<<<<<<] >>>>>>>-]>> propagate n <br />
[<[->+<]>>]<. sum n copies of m
</div>
<br />
<div class="myaux">
Note that, for brevity, I have limited one of the operands to 5, however the expansion to an arbitrary number would be straightforward by increasing all stretches of <span class="mycode">>>>>>>></span> and increasing the number of replications from 5 to the desired maximum.
</div>
<br />
You can test it in <a href="https://repl.it/repls/SpryOverjoyedAccess">a BrainF emulator</a> for yourself that it works (screenshot).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCTD4JYt7NtVfC3Up6BSSEZ7OyXCiRMjztntCKqw2ERHspDKYI7K8w5Xgcs-Fh8wqyL8djY4eXHoGNkVGSHApLSPm5NHu0rsjNDAekcYSUSe6i-HxrFmc7zh44-1iV7BKJB4M6mWGNjrA/s1600/BrainFdemo.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="155" data-original-width="904" height="109" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCTD4JYt7NtVfC3Up6BSSEZ7OyXCiRMjztntCKqw2ERHspDKYI7K8w5Xgcs-Fh8wqyL8djY4eXHoGNkVGSHApLSPm5NHu0rsjNDAekcYSUSe6i-HxrFmc7zh44-1iV7BKJB4M6mWGNjrA/s640/BrainFdemo.PNG" width="640" /></a></div>
<br />
For debugging you can use something like <a href="https://fatiherikli.github.io/brainfuck-visualizer/">this visualizer</a>, but note that it's heavily memory limited.<br />
<br />
The real surprise was when I decided to compare my solution against the internet, dimly suspecting that there may be far superior alternatives out there. And indeed, as <a href="https://stackoverflow.com/questions/5165772/code-for-multiplying-two-one-digit-numbers-in-brainfuck">this StackOverflow discussion</a> nicely points out, there is a MUCH more elegant code doing this. Here goes,<br />
<div class="mycode">
[<br />
>[>+>+<<-]<br />
>[<+>-]<br />
<<-]
</div>
<br />
Compared to this, my example above looks like such baaaad code :)
<br />
<div class="myaux">
What has this taught me? Several things:<br />
<ul><li>
Brainf*ck can be used as a measure of one's "internal memory capacity" for solving puzzles and math problems. Mine turns out to be pretty limited. I simply cannot, at least not quickly and entirely in my mind, come up with a program that requires you to keep track of more than 3 conditions at once (like 3 loops). <span class="myinline">Well, I kinda knew this already from my school-time profound hate of learning poems by heart.</span><br />
</li><li>On the other hand, I was able to come up with a working solution, all by myself, in circumstances that were very very far from optimal.<br />
</li><li>Learning curve is very important. Even in Brainf*ck, there are typical nuts-and-bolts programming tricks that make life so much easier once you've mastered them. I guess that just as I found it very difficult to stop thinking in higher-level languages such as (at least) the assembly, many people who never coded in their life would find it very difficult to grasp what we almost take for granted, such as <span class="mycode">i=i+1</span>, or for-loops, or pointer arithmetic, or polymorphism, or templates.<br /></li></ul>
<br />
<b>I hope this exercise will make it a bit easier for me to explain programming to non-programmers.</b><br />
</div>
<br />
<br />Sergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-76469758531318142762019-06-05T16:09:00.000-04:002019-12-13T15:04:10.155-05:00Excel VBA: Automatic macro execution for Windows Task Scheduler (and a few pranks to boot)Despite all modern trends towards workflow automation, manual Excel-based workflows still abound in many areas of industry, notably finance. In <a href="https://curiousercode.blogspot.com/2019/03/excel-vba-automatic-opening-of-external.html">our previous post</a> we learned how to simplify and automate some of them, reducing manual copy-and-paste to a button push.<br />
<br />
The next step of this would be further automation, so that the"button" gets pushed automatically, for example on a daily basis at a predetermined time.<br />
<br />
One intuitive approach would be to invoke your code using an <a href="https://www.automateexcel.com/vba/auto-open-macro/">Auto Macro</a> such as <span class="mycode">Workbook_Open</span>, and then have <a href="https://docs.microsoft.com/en-us/windows/desktop/taskschd/task-scheduler-start-page">Windows Task Scheduler</a> open your workbook on schedule as required.<br />
<br />
However, there is a catch here. Workbooks used for automated workflow are very often <i>also</i> used for ad-hoc manual tasks, or are simply opened to view some data, with no intention of performing any (re)calculations. Now your auto macro code would run <i>every</i> time the workbook is opened. If the automated workflow involves some data modification and/or email notification, you will have side effects very time you open your workbook just to see what is in there.<br />
<br />
Worse still, if your workbook is on a shared drive, other used may stumble upon it by accident, triggering execution of the auto macros when they never expect it.<br />
<br />
Of course, you could run two copies of your spreadsheet - one for automation and one for manual inquiries, naming the "automation" one something like <span class="mycode">"never_ever_open_this_manually.xlsm"</span>, but what happens in practice in such cases is that the two workbooks fall out of sync, and one of them is always out of date (or worse, they are both partially out of date).<br />
<br />
A much more elegant solution would be for the auto macro to <i>detect</i> that the workbook was invoked by the Task Scheduler and not manually, and run the workflow only in this case.<br />
<br />
After some probing and <a href="https://stackoverflow.com/questions/27293802/is-it-possible-to-run-macro-only-when-excel-file-is-opened-by-task-scheduler">looking for inspiration</a> I have come up with this solution:<br />
<br />
<pre class="brush:vb">Private Function IsScheduled() As Boolean
' Determine if the workbook was opened manually or as part of a Task Scheduler routine
On Error GoTo Decided
Dim Result As Boolean
Result = True ' An invisible or programmatically started session is always assumed scheduled
If Not Application.Visible Then GoTo Decided
If Not Application.UserControl Then GoTo Decided
Result = False ' A session with more that one workbooks is always assumed manual
If Application.Workbooks.Count > 1 Then GoTo Decided
' otherwise assume scheduled task if the workbook name was supplied in the command line
Dim wname As String, cmdline As String
cmdline = UCase(Trim(GetCommandLine)): wname = UCase(Trim(ThisWorkbook.Name))
Result = InStr(cmdline, wname) > 0
Decided: IsScheduled = Result
On Error GoTo 0
End Function
Private Sub Workbook_Open()
If Is Scheduled Then
' Run your automated workflow here
End If
End Sub
</pre>
<br />
It basically relies on the discovery that if a workbook is started by the Task Scheduler, its command line would be <br />
<div class="mycode">
C:\Program Files\OFFICE##\Excel.exe v:\path_to_workbook\workbook.xlsm</div>
whereas for most other scenarios of workbook opening (directly from Excel or from Windows Explorer) the command line would look like
<br />
<div class="mycode">
C:\Program Files\OFFICE##\Excel.exe</div>
or <br />
<div class="mycode">
C:\Program Files\OFFICE##\Excel.exe /dde</div>
i.e. would not contain the workbook name (instead, it passes the workbook via <a href="https://en.wikipedia.org/wiki/Dynamic_Data_Exchange">DDE</a>)<br />
<br />
To get to the command line, we need to implement the following API function (<a href="https://stackoverflow.com/questions/47757827/excel-64-bit-command-line-vba-code">taken from here</a>):<br />
<br />
<pre class="brush:vb">#If Win64 Then
Private Declare PtrSafe Function GetCommandLineL Lib "kernel32" Alias "GetCommandLineA" () As LongPtr
Private Declare PtrSafe Function lstrcpyL Lib "kernel32" Alias "lstrcpyA" (ByVal lpString1 As String, ByVal lpString2 As LongPtr) As Long
Private Declare PtrSafe Function lstrlenL Lib "kernel32" Alias "lstrlenA" (ByVal lpString As LongPtr) As Long
#Else
Private Declare Function GetCommandLineL Lib "kernel32" Alias "GetCommandLineA" () As Long
Private Declare Function lstrcpyL Lib "kernel32" Alias "lstrcpyA" (ByVal lpString1 As String, ByVal lpString2 As Long) As Long
Private Declare Function lstrlenL Lib "kernel32" Alias "lstrlenA" (ByVal lpString As Long) As Long
#End If
Private Function GetCommandLine() As String
GetCommandLine = "FAILED TO RETRIEVE" ' fallback value is case of error
On Error GoTo Finally ' suppress errors
Dim strReturn As String
#If Win64 Then
Dim lngPtr As LongPtr
#Else
Dim lngPtr As Long
#End If
Dim StringLength As Long
'Get the pointer to the commandline string
lngPtr = GetCommandLineL
'get the length of the string (not including the terminating null character):
StringLength = lstrlenL(lngPtr)
'initialize our string so it has enough characters including the null character:
strReturn = String$(StringLength + 1, 0)
'copy the string we have a pointer to into our new string:
lstrcpyL strReturn, lngPtr
'now strip off the null character at the end:
GetCommandLine = Left$(strReturn, StringLength)
Finally: On Error GoTo 0
End Function
</pre>
<br />
Note the compiler directives to make the code 32/64-bit aware -- this is absolutely essential if the workbook resides on a shared drive in a network with mixed 32/64-bit Excel installation. Otherwise opening the workbook might crash the entire Excel session for random unsuspecting users.<br />
<br />
<br />
<div class="myaux">
BONUS: When debugging the automation routine I have stumbled upon some wonderful tools you can use to play a practical joke on your peers. Namely you can use <a href="https://docs.microsoft.com/en-us/office/vba/api/excel.application.onkey"><span class="mycode">Application.OnKey</span></a> to override the function of a commonly used key (like, a letter or an arrow) to something totally bizarre, and wrap it into <a href="https://docs.microsoft.com/en-us/office/vba/api/excel.application.ontime"><span class="mycode">Application.OnTime</span></a> to activate at some (possibly random) point in the future. Put it in your auto macro, like so: <br />
<br />
<pre class="brush:vb">Public num As Integer
Function sov(sekunder As Double) As Double
starting_time = Timer
Do
DoEvents
Loop Until (Timer - starting_time) >= sekunder
End Function
Sub GetDizzy()
num = num + 1
On Error Resume Next
Select Case (num Mod 5)
Case 2 'move to opposite drection
ActiveCell.Offset(0, 1).Select
Case 3 'shake the workbook
dx = Round(Rnd * 10) - 5: dy = Round(Rnd * 10) - 5
For i = 1 To 3
ActiveWindow.SmallScroll toright:=dx, down:=dy: sov (50 / 1000)
ActiveWindow.SmallScroll toleft:=dx, up:=dy: sov (50 / 1000)
Next i
Case 4 'nudge the workbook
ActiveCell.ColumnWidth = ActiveCell.ColumnWidth + Round(Rnd * 15) - 7
Case Else 'normal operation
ActiveCell.Offset(0, -1).Select
End Select
If num > 200 Then MsgBox "You are tired and dizzy... Why don't you call it a day and go home?"
On Error GoTo 0
End Sub
Sub Payload()
Application.OnKey "{LEFT}", "GetDizzy"
End Sub
Private Sub Workbook_Open()
Application.OnTime Now + TimeValue("8:00:00"), "Payload"
End Sub
</pre>
<br />
and watch your colleagues of choice get delighted at the emphatic computer offering them to go home at the end of the working day because they are tired and have worked long hours and it's late outside (I've borrowed the sleep function <span class="mycode">sov()</span> <a href="https://stackoverflow.com/a/45729870">from here</a>). The code is very hard to detect because it executes totally silently and persists even after the workbook containing has long been closed; the empathy will last until Excel is closed or restarted -- but in most industry workplaces I have seen, this only happens when it crashes.<br />
<br />
<span class="myinline">Your mileage may actually vary with the payload, but do remember not to use it for any real sabotage because this is a surefire way to get fired, if you excuse the tautology again.</span></div>
<br />
<br />Sergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-44671188683294481152019-03-05T18:00:00.000-05:002019-03-05T18:00:11.456-05:00Excel VBA: Automatic opening of external files and other neat tricksSuppose you have a workflow in Excel that involves manual copy-and-paste of data from one spreadsheet (A) into another spreadsheet (B), and you want to automate it.<br />
<br />
In some cases, you might make use of data sources and automatic linking between files, but there are many cases where this would be undesirable. Either the automatic update of links makes your process undesirably slow, or you need those intermediate values unchanged for reporting/auditing purposes, or the links update is just too buggy and unreliable (as are, alas, many automatic features in the MS Office suite), whatever the reason, sometimes you are better off with the "plain old-school" VBA macro approach, approximately,<br />
<br />
<ol>
<li>Open the external spreadsheet A</li>
<li>Find the source data</li>
<li>Copy the data into your destination spreadsheet B</li>
</ol>
<div>
To do this gracefully, one must think through several scenarios though. What if the source spreadsheet is already open - should we re-open it (possibly discarding unsaved changes), or should we reuse the already opened sheet? What do we do after the data copying is done - should we force-close the source sheet or should we keep it? </div>
<div>
<br /></div>
<div>
After giving these questions some thought, I have created a short VBA code snippet, and after having re-used it about a dozen times over the course of less than one month, I am putting it here for reference. Here goes:</div>
<div>
<br /></div>
<div>
<pre class="brush: vb">
Set targetWB = Application.ActiveWorkbook
TargetPath = targetWB.Path & "\"
SourcePath = "..\otherfolder" 'Relative to target workbook
SourceName = "somename.xlsx" 'Can be any calculation to determine name
alreadyOpen = False
For Each thisWB In Workbooks
If thisWB.Name = SourceName Then
alreadyOpen = True: Set sourceWB = thisWB
End If
Next thisWB
On Error GoTo ERRORHANDLER ' to gracefully handle "File not found" scenarios
If Not alreadyOpen Then Set sourceWB = Workbooks.Open(Filename:=TargetPath & SourcePath & SourceName, ReadOnly:=True)
On Error GoTo 0
' *** Data copying code goes here ***
If Not alreadyOpen Then sourceWB.Close SaveChanges:=False
Set sourceWB = Nothing: Set targetWB = Nothing ' garbage collection
</pre>
</div>
<div>
<br /></div>
<div>
Note a few touches here: </div>
<div>
<ul>
<li>The code reuses the already opened spreadsheet if it is already open (pardon the tautology), and opens it otherwise. This makes the code both faster for debugging (no repeated open/close - the source workbook can be large!) and suitable for automated production workflows. The only downside here is that you need to separate the actual name of the spreadsheet, and a path to it; however if you only have the full path, you can easily work around it with a one-liner like so:
<pre class="brush: vb">SourceName = FullPathName: While InStr(SourceName,"\") > 0: SourceName = Right(SourceName,Len(SourceName)-InStr(SourceName,"\")): Wend</pre>
using <span class="mycode">FullPathName</span> instead of <span class="mycode">TargetPath & SourcePath & SourceName</span> in <span class="mycode">Workbooks.Open(...)</span>.</li>
<li>The code automatically closes the source spreadsheet only if it had to open it. This way, no spurious workbooks are left open in a production workflow.</li>
<li>The files are opened read-only and closed without saving changes. This prevents unwanted queries that can otherwise pop up if the file happens to be opened by another user, or if the data copying process ends up messing up the source file (which, depending on how complicated the data processing routine is, is quite likely to happen). These queries are annoying whilst debugging and totally disruptive in a production workflow; last but not least we are preventing unwanted modification of the source file.</li>
</ul>
</div>
<div>
<br /></div>
<div>
<br /></div>
<br />
<br />
<div class="myaux">
BONUS: Here is the routine I commonly use for the actual data copying. I find it preferable to using selection / clipboard / autofilter operations because it involves less GUI interaction and therefore is more robust (and, with <span class="mycode">Application.ScrenUpdating=True</span>, can be faster!)<br />
<pre class="brush: vb">
targetWS = "Analysis" : sourceWS = "Data" 'use any names
targetStartAt = "A2": sourceStartAt = "A2"
'feel free to use named ranges here
'or to have your code determine the locations based on search criteria
Set here = sourceWB.Worksheets(sourceWS).Range(sourceStartAt)
Set there = targetWB.Worksheets(targerWS).Range(targetStartAt)
While Len(here.value) > 0
If Not IsError(here.Value) Then
If here.Value = "GOOD" Then ' put whatever condition to validate a source line to determine if it needs copying
' example of data copying, edit as your task requires
there.Value = here.Value
there.Offset(0,1).Value = Left(here.Offset(0,1).Value,8)
For i = 4 to 12
there.Offset(0,i-2).Value = here.Offset(0,i).Value
Next i
Set there = there.Offset(1,0)
End If
End If
Set here = here.Offset(1,0)
Wend
</pre>
</div>
<br />
<br />Sergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-64326471830263320982019-02-06T14:32:00.000-05:002019-02-22T16:03:14.332-05:00No more calculation babysitting!Let's imagine you work with a computer on a daily basis (which, as it were, is a rather common scenario these days), and let's imagine your work includes "relatively lengthy" computational tasks -- lengthy enough that it is not productive to just sit there staring at the proverbial hourglass cursor (or meditating over the progress bar, or praying over the build output console window that it compiles without errors, or whatever it is). So...<br />
<br />
So, after some time waiting (depending on your boredom threshold, about 30 seconds for me, I'd guess under 2 minutes for most people) you decide to multitask and switch to another task (work-related or otherwise). Before long, that other task immerses you and you realize that your lengthy computation was actually done minutes ago and you could have, and should have, resumed your workflow earlier. So in an attempt to avoid doing nothing and increase productivity, you just dumped it down the tubes.<br />
<br />
Or consider another scenario. Your computation is now lengthy enough (say, 10-20 minutes) so that you decide to grab a coffee from the kitchen, or grab a quick bite/smoke/chat, or whatever it is. It would be cool if you could get an alert that your task has finished, so you can timely return and resume work without having to check on your workstation multiple times.<br />
<br />
Or yet another scenario. You need to run an even lengthier calculation, perhaps a few hours on end, so you leave it running after hours. And you have several of them to run after each other, and you want to run as many as you can before the next working day. So again it would be cool if you can get an alert once your calculation finishes, rather than having to log in and check on the computation progress repeatedly. More importantly, if your task is aborted early due to some mishap (which happened to me due to my own banana fingers more times than I'd confess), you really want to be alerted at once, rather than after what you thing the task should have taken, had it completed normally.<br />
<br />
The first scenario can partially be mitigated by running Windows Task Manager, minimizing it, and noticing when the CPU usage drops, indicating that your task has finished. But you still need to remain vigilant and keep watching that tiny indicator, and with current multicore CPUs, the drop may be from 13% to some 2%, which is not very noticeable visually. Not to mention that the two remaining scenarios cannot be worked around in this way.<br />
<br />
Wouldn't it be nice to have an automated monitor which can do this for you?<br />
<br />
Yeah it sure would.<br />
<br />
So let's see how we can do it in Windows Powershell. You can determine the CPU usage of a process using this function (loosely adapted from <a href="https://www.petri.com/powershell-problem-solver-process-cpu-utilization" target="_blank">here</a>):<br />
<br />
<pre class="brush:ps">function get-excel-CPU ($avgs=1)
{
$result=0
for ($i=1; $i -le $avgs; $i++)
{
$cpuinfo = Get-WmiObject Win32_PerfFormattedData_PerfProc_Process -filter "Name LIKE '%EXCEL%'"
$result=$result + ($cpuinfo.PercentProcessorTime |Measure -max).Maximum
start-sleep -m 150
}
$result = $result/$avgs
$result
}
</pre>
<br />
<div class="myaux">
Note the following:<br />
<br />
<ul>
<li>I use EXCEL as an example because I use it most often. It is trivial to modify it to work with any other program.</li>
<li>The function takes into account that there can be multiple instances of your program, and will report the CPU usage of the most CPU intensive process. Usually, this is what you want, because only one of your instances will be doing computations anyway, but you can easily fine-tune it to be more instance-specific.</li>
<li>The function measures CPU usage several (<span class="mycode">$avgs</span>) times with a short waiting period in between, and then averages the measurement. This is done because some computations, mostly ones heavily on local or network I/O, will have wildly fluctuating CPU usage, so taking one measurement may trick the script into falsely deciding that the computation has finished. You may need to fine-tune the number of measurements and the wait time between them to reflect your specific computation pattern.</li>
</ul>
</div>
<br />
Now that we have a way to automatically determine the CPU usage, we can easily wrap it in a control loop:<br />
<br />
<pre class="brush: ps">$poll = 10
$homethresh = 600
$sensitivity = 5
$fullout = new-timespan -Seconds 86400
$mailout = new-timespan -Seconds $homethresh
$sw = [diagnostics.stopwatch]::StartNew()
$thiscpu = get-excel-cpu 5
if ($thiscpu -lt $sensitivity)
{
write-host $thiscpu, ": Excel not running, exiting."
}
else
{
write-host "Initial CPU is ", $thiscpu
$cnt=0
while ($sw.elapsed -lt $fullout)
{
start-sleep -s $poll
$thiscpu = get-excel-cpu 3
$lock = is-locked
write-host "Elapsed", $sw.elapsed, " -- CPU is ", $thiscpu, " ",$lock
if ($thiscpu -lt $sensitivity) {$cnt++} else {$cnt=0}
if ($cnt -ge 2)
{
write-host "FINISHED!!!!"
if ($lock -eq "UNLOCKED") {show-splash} else {phone-home}
return
}
}
write-host "Timed out!"
}
</pre>
<br />
Note the line <span class="mycode">if (...) {show-splash} else {phone-home}</span> . There are two ways to alert you that the computation has finished. One is to show you a big splash screen, borrowed from <a href="https://blogs.technet.microsoft.com/stephap/2012/04/23/building-forms-with-powershell-part-1-the-form/" target="_blank">here</a>:<br />
<br />
<pre class="brush: ps">function show-splash
{
Add-Type -AssemblyName System.Windows.Forms
$Form = New-Object system.Windows.Forms.Form
$Form.Text = "Finished"
$Form.AutoSize = $True
$Form.AutoSizeMode = "GrowAndShrink"
$Form.BackColor = "Lime"
$Font = New-Object System.Drawing.Font("Arial",96,[System.Drawing.FontStyle]::Bold)
$Form.Font = $Font
$Label = New-Object System.Windows.Forms.Label
$Label.Text = "Calculation finished!"
$Label.AutoSize = $True
$Form.Controls.Add($Label)
$Form.Topmost=$True
# -- this ensures your splash screen appears on top of other windows!
$Form.ShowDialog()
}
</pre>
<br />
The other is to simply send you an email that gets pushed to your smartphone or smart watch, borrowed from <a href="https://www.computerperformance.co.uk/powershell/function-send-email/" target="_blank">here</a> <span class="myinline">(using Outlook rather than <span class="mycode">Send-MailMessage</span> so that your IT department can safely inspect your outgoing email and won't mistake your script for a trojan):</span><br />
<br />
<pre class="brush: ps">function phone-home
{
$Outlook = New-Object -ComObject Outlook.Application
$Mail = $Outlook.CreateItem(0)
$Mail.To = "youremailaddress@mailserver.com"
$Mail.Subject = "Calculation finished"
$Mail.Body ="Your calculation has finished. If you need to start another one, go for it."
$Mail.Send()
}
</pre>
<br />
Now, how to choose between the two? You will want the splash screen if you are sitting in front of your screen, and the email otherwise. So we need a way of discriminating between the two. Following <a href="https://powershell.org/forums/topic/how-to-find-computer-screen-is-locked-or-not-using-powershell/" target="_blank">this idea</a>, we can use<br />
<pre class="brush: ps">function is-locked
{
try {
$currentuser = gwmi -Class win32_computersystem | select -ExpandProperty username
$process = get-process logonui -ea silentlycontinue
if($currentuser -and $process){"LOCKED"}else{"UNLOCKED"}
return}
#Always return LOCKED if logged in remotely
catch{"LOCKED";return} }
</pre>
<br />
Finally, here is a BAT-file one-liner wrapper, called <span class="mycode">ps.bat</span> to run your PowerShell script on systems where execution of random scripts has been disallowed by default (for good reason). We cannot override this default without admin privileges, but we can by pass it temporarily by calling<br />
<br />
<pre class="brush: ps">@powershell -ExecutionPolicy RemoteSigned .\%1.ps1
</pre>
<br />
You can then call your PowerShell script, e.g., <span class="mycode">poll.ps1</span>, and simply type <span class="mycode">ps poll</span> in your command prompt to invoke it quickly.<br />
<br />
Enjoy!Sergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-74915787438112531072019-02-01T17:01:00.000-05:002019-02-01T17:01:22.769-05:00Electric cabinet lock and other small DIY<div dir="ltr" style="text-align: left;" trbidi="on">
Every once in a while, we have people (such as cleaners, babysitters or contractors) who have access to our home while we are away or distracted. And I am perfectly aware that petty theft usually does not happen in these scenarios (one case is enough to ruin the perpetrator's career), we also know that from time to time, against all odds, it <i>does</i> happen. So we needed to come up with an idea of protecting my wife's jewelries from an opportunistic snatch.<br />
<div>
<br /></div>
<div class="myaux">
In other words, I needed a lock on the jewelry drawer, preferably one that would be discreet, and would not involve drilling the front of the (moderately beautiful) dresser cabinet. </div>
<div>
<br /></div>
<div>
In a previous edition of this scenario, I accomplished this using two magnetic child locks (like <a href="https://www.amazon.ca/Safety-1st-Magnetic-Cabinet-Locks/dp/B004GCJMLG" target="_blank">these ones</a>, they come in a huge variety) - to open the drawer you'd need to simultaneously place magnetic keys at two unmarked, previously known spots, which makes for an excellent discrete opening mechanism. But this time the cabinet dimensions proved incompatible with the locks. To add to that, even a casual glimpse of the open drawer, sporting the big white child locks, would instantly reveal our trick. Finally, the mounting holes for magnetic locks look bad even on the inside of the cabinet. <span class="myinline">(Whoever tries to convince you that these locks would hold on an adhesive mount, has not tried it. Adhesive mounting tape can be strong, but is is invariably terrible for dynamic loads such as repeated banging from trying to open a cabinet with the lock engaged but forgotten about.)</span></div>
<div>
<br /></div>
<div>
So I was searching for a geometrically suitable lock and stumbled upon <a href="https://www.amazon.ca/gp/product/B00YRASLB4/" target="_blank">this part</a>, which just happened to have just the right dimensions to fit between the back wall of the drawer and the back wall of the cabinet. In addition, an electromagnetic lock has the advantage of not requiring submillimeter-precision alignment between the lock and the armature/striker plate - something that would totally plague most mechanical designs (like <a href="https://www.amazon.ca/gp/product/B078SQGCN9" target="_blank">this one</a>). </div>
<div>
<br /></div>
<div>
I already had an unused <a href="https://www.amazon.ca/gp/product/B007QAJ2Q0" target="_blank">electric key switch</a>, which could be discretely built into the back wall of the cabinet, totally out of sight yet within easy reach. The only missing piece now was how to power the lock. Using a DC power adapter seemed an easy choice, but it is very easily defeated by unplugging it from the wall. Using batteries (and placing them in the same space between the drawer and the cabinet) was more secure, but with the lock's power consumption of 100 mA, batteries would require replacement every 1-2 days. </div>
<div>
<br /></div>
<div>
Therefore, an ideal trade-off would be a plugged in adapter with a battery back-up that would take over the lock if the adapter is unplugged. After giving it some thought I came up with this circuit:</div>
<div>
<br /></div>
<div>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5RS33afQUaAGFFGc78d684xXPi9erXXxPSKYLpLvPk_mNCh-2U8GM76RVjM-40_-43YAAn3qXIKdkijJN3G4A0NOICnvNIcrAFmy6nN1Wxmzhk2GWi9ugmq432Z8KI2qQgVQkc5101jA/s1600/lock-final.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="251" data-original-width="414" height="194" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5RS33afQUaAGFFGc78d684xXPi9erXXxPSKYLpLvPk_mNCh-2U8GM76RVjM-40_-43YAAn3qXIKdkijJN3G4A0NOICnvNIcrAFmy6nN1Wxmzhk2GWi9ugmq432Z8KI2qQgVQkc5101jA/s320/lock-final.png" width="320" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
The central component here is the relay K1 that connects the battery through the normally closed contacts and disconnects it when external DC power is present. The diode D1 is there to ensure that the battery is not getting charged by the adapter (non-rechargeable batteries don't like that). The diode D2 ensures that the battery is not getting discharged through the output circuitry of the adapter. The diodes D3 and D4 are <a href="https://en.wikipedia.org/wiki/Flyback_diode" target="_blank">flyback diodes</a>, which prevent arcing and sparking at the key switch (or the power jack).<br />
<br />
<div class="myaux">
Finally, the resistor R1 is the current limiting resistor for the relay coil. It actually proved the most problematic component because of the need to mitigate heat dissipation in the enclosed space behind the cabinet. With some experimenting I found that the relay coil current for reliable operation was 40 mA, yielding 0.32 W of dissipated heat, so when I stupidly put a 0.125W resistor, it got pretty charred very soon. Even a 0.5W resistor was getting worryingly hot. Since I totally don't want my lock to start a house fire - that would be the exact opposite of what a "security lock" is supposed to do - I first hooked up four 800 Ohm, 0.5W resistors in parallel (2W total). This got the resistors slightly warm to the touch but not hot. Here is how it looks like from the inside and outside:<br />
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEilw_eW2oPP5q-Cnf8RgqxASHn9wCzR7siWIVyZyEDU84kBcEBq4bQ8wJnjk3CqfwvUEv4-Y714H_ANX5JbVGJKeQWQvG9uMyzbe8JFBGmUrnpKK0j0CCyW7tLmG3kARFUN0hkFbmoyQXM/s1600/foto_no_exif+%25281%2529.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="778" data-original-width="1600" height="155" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEilw_eW2oPP5q-Cnf8RgqxASHn9wCzR7siWIVyZyEDU84kBcEBq4bQ8wJnjk3CqfwvUEv4-Y714H_ANX5JbVGJKeQWQvG9uMyzbe8JFBGmUrnpKK0j0CCyW7tLmG3kARFUN0hkFbmoyQXM/s320/foto_no_exif+%25281%2529.jpg" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAJLZLbZ50_4DvmSQH3Xhpkpl15ArBP5xCF-Uy7qZwWuA2HroW0eGBcAaIdGrm0_lmM_u-bDqjPE6XnPovkhyuR1eLk4tuRaWT5E685noCbmbk1bSF65rT77d3KiQ_JoWddAlpkt3Qj4I/s1600/foto_no_exif+%25282%2529.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="778" data-original-width="1600" height="155" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAJLZLbZ50_4DvmSQH3Xhpkpl15ArBP5xCF-Uy7qZwWuA2HroW0eGBcAaIdGrm0_lmM_u-bDqjPE6XnPovkhyuR1eLk4tuRaWT5E685noCbmbk1bSF65rT77d3KiQ_JoWddAlpkt3Qj4I/s320/foto_no_exif+%25282%2529.jpg" width="320" /></a></div>
<br />
<br />
As an upgrade, I later replaced the 2W resistor arrangement with a 5W component on a heat sink. Now operating at 6% capacity, the heat dissipation was small enough for the lock to run for days on end without getting warm. As an additional precaution, I have installed a thermal fuse designed to cut the AC power circuit should the temperature ever exceed 73 degrees C (this was the lowest value I could get off Amazon). So this is the new set-up:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUm7ZpzP1PEY1SuWsiRrHqaDkzqih70v_tPvlWyhDthCjS02XCu81HkEr7JCdZppu5FiDAiKWMb6T7ekLUQ0ho7ZEQfCLTmf5SKL0-Fvk4TbJskBNvUqNLSemTFv-MHwfLKzcS_v629wE/s1600/foto_no_exif.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="778" data-original-width="1600" height="155" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUm7ZpzP1PEY1SuWsiRrHqaDkzqih70v_tPvlWyhDthCjS02XCu81HkEr7JCdZppu5FiDAiKWMb6T7ekLUQ0ho7ZEQfCLTmf5SKL0-Fvk4TbJskBNvUqNLSemTFv-MHwfLKzcS_v629wE/s320/foto_no_exif.jpg" width="320" /></a></div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
The beauty of the design, aside from it being totally discrete, is its being <i>both </i>fail-secure <i>and</i> fail-safe. It is fail-secure in the short term, meaning that power outage will keep the lock running on battery power long enough for a malefactor to not want to stick around. On the other hand, the rightful owner can wait several days for the batteries to discharge, and have the cabinet open if the key gets misplaced or lost.<br />
<br />
<br />
<div class="myaux"> BONUS: Here's some more office DIY. At one of my workplaces, the desk phone used too much useful space on my desk, so I wanted it next to my desk instead. Not wanting to drill any holes in the shiny new company property, I came up with a mount out some stuff lying around in the office, namely:<br />
<br />
<ul>
<li>an old cardboard small packet from a recent online order</li>
<li>some Scotch tape</li>
<li>some good supply of cable ties</li>
<li>and a jar of spare furniture bits and pieces, apparently mostly from IKEA. </li>
</ul>
This is the "before" and "after" image. If interested, I can give you more detail.</div>
<br />
<br />
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdXEWnmxE5a_BK_nOC4skiyCBHKNX92MkjCx9uRhWbk3Cg5dmGbW8NOgv0HoFXnIfIDuVDXfUj4ZTNAMUxfVPo9ehdYaII9FzLNA-4_z7QC-gfqJ3w79J9gil3Ld80VZfpX5MLUJXXzsQ/s1600/20180611_164906.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdXEWnmxE5a_BK_nOC4skiyCBHKNX92MkjCx9uRhWbk3Cg5dmGbW8NOgv0HoFXnIfIDuVDXfUj4ZTNAMUxfVPo9ehdYaII9FzLNA-4_z7QC-gfqJ3w79J9gil3Ld80VZfpX5MLUJXXzsQ/s320/20180611_164906.jpg" width="240" /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjH3BVk9GNLm7J1BqbiJyp_1P5mcbXpGeprMU4Voc-NtN1P6UAimIidmQ6NqrGJFVz1rELfk7a4HupFCSGGh498bSjKTfJiTUVdu1TEYqgmT7aBu0W4Yd9FZ_MI6qUNVhEUOjQxgFk5Q2g/s1600/20180611_164952.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjH3BVk9GNLm7J1BqbiJyp_1P5mcbXpGeprMU4Voc-NtN1P6UAimIidmQ6NqrGJFVz1rELfk7a4HupFCSGGh498bSjKTfJiTUVdu1TEYqgmT7aBu0W4Yd9FZ_MI6qUNVhEUOjQxgFk5Q2g/s320/20180611_164952.jpg" width="320" /></a></div>
<br /></div>
</div>
Sergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-74202539839629725412018-11-20T16:45:00.000-05:002019-12-13T15:04:22.175-05:00Memory leak tester in Excel<div dir="ltr" style="text-align: left;" trbidi="on">
In many scenarios involving complicated Excel calculations, especially those relying on extensive VBA code and/or custom-made add-ins, there is a danger that memory leaks could be created by programming errors such as careless object operations.<br />
<div>
<br /></div>
<div>
Most often, such memory leaks are discovered <i>post factum</i>, when, left unchecked, they cause slowdowns or crashes when an Excel session runs out of memory. In such a case, correcting the error becomes a major pain. How to spot a memory leak offender in a calculation intensive spreadsheet containing thousands of custom functions? </div>
<div>
<br /></div>
<div>
To this end, I have written a very simple program that basically evaluates a given formula multiple times and measures how the memory consumption increased. I have used this googled snippet to determine the memory consumption:</div>
<div>
<br /></div>
<div>
<pre class="brush:vb">Declare Function GetCurrentProcessId Lib "kernel32" () As Long
Function GetMemUsage()
' Returns the current Excel.Application memory usage in KB
Set objSWbemServices = GetObject("winmgmts:")
GetMemUsage = objSWbemServices.Get( _
"Win32_Process.Handle='" & _
GetCurrentProcessId & "'").WorkingSetSize / 1024
Set objSWbemServices = Nothing ' We don't want to cause memory leaks here :)
' We don't want to cause memory leaks here :)
End Function
</pre>
</div>
<div>
<br /></div>
<div>
and wrote a very simple wrapper: </div>
<div>
<br /></div>
<div>
<pre class="brush:vb">Sub Measure_Leak()
Set here = ActiveSheet.Range("A2")
template = here.Value
i1from = here.Offset(0, 1).Value: i1to = here.Offset(0, 2).Value
i2from = here.Offset(0, 3).Value: i2to = here.Offset(0, 4).Value
Set there = here.Offset(0, 5)
there.Offset(0, 1).Value = GetMemUsage()
For i1 = i1from To i1to
For i2 = i2from To i2to
working = "=" & template
On Error Resume Next
working = Replace(working, "$1", i1)
working = Replace(working, "$2", i2)
On Error GoTo 0
there.Formula = working
Next i2, i1
there.Offset(0, 2).Value = GetMemUsage()
Set here = Nothing
Set there = Nothing
ActiveSheet.Calculate
End Sub
</pre>
</div>
<div>
<br /></div>
<div>
Now I put this on an even simpler spreadsheet which looks like this:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQOW5CURYVU0f7FSzEM3qTcnIdrEu6LYIEszbsYMxVKq8uqEmI9t5HjxrrEjMPh7FyvcSmjQrYxEMj7UJxOnfchBJb4vSA3M-tnP7J4Wh71Y46Kh2H2FQxUA97lLNjBx5YYn4GMNAexZ8/s1600/memleak.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="195" data-original-width="1027" height="121" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQOW5CURYVU0f7FSzEM3qTcnIdrEu6LYIEszbsYMxVKq8uqEmI9t5HjxrrEjMPh7FyvcSmjQrYxEMj7UJxOnfchBJb4vSA3M-tnP7J4Wh71Y46Kh2H2FQxUA97lLNjBx5YYn4GMNAexZ8/s640/memleak.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<div class="myaux">
Basically, pressing the <b>Measure</b> button evaluates the template expression substituting <span class="mycode">$1</span> and <span class="mycode">$2</span> with values spanning two ranges, a total of <span class="mycode">(i2t-i2f+1)*(i1t-i1f+1)</span> times. An increased memory footprint at the end of the execution means there is a memory leak. </div>
<br />
In the screenshot we see that a built-in Excel function does not cause any memory leaks (hurra!)</div>
<div>
<br /></div>
<div>
To test it, let us define a really leaky VBA function using <a href="http://www.vbi.org/Items/article.asp?id=106" target="_blank">this example</a>:</div>
<div>
<br /></div>
<div>
<pre class="brush:vb">' Put this into class module Class1
Option Explicit
Private A As New Class2
Private Str As String
Private Sub Class_Initialize()
Set A.B = Me ' Fool garbage collector
Str = Space(1024 * 10) ' Allocate lots of memory
End Sub
'Put this into class module Class2
Option Explicit
Public B As Class1
</pre>
</div>
<div>
<br /></div>
<div>
with two functions that look similar but one is known to be leaky:</div>
<div>
<br /></div>
<div>
<pre class="brush:vb">Function vbaNoLeak()
Dim MyObject As Class2
Set MyObject = New Class2
Set MyObject = Nothing
End Function
Function vbaLeak()
Dim MyObject As Class1
Set MyObject = New Class1
Set MyObject = Nothing
End Function
</pre>
</div>
<div>
<br /></div>
<div>
...and...</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiL5KUNhdQPCra7W7wTfkyMQZ8D6hqGttvlg4Q_nzEnmmpXidR5y58dg1wfpTwb35R6TADBnIIVaOlk2NCE_4H0D5ALycs1Mcnk_xqD5x7fJVO-Zd4WT1JuOjWjCae_acDidXvzQpIS81E/s1600/memleak-NO.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="164" data-original-width="1021" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiL5KUNhdQPCra7W7wTfkyMQZ8D6hqGttvlg4Q_nzEnmmpXidR5y58dg1wfpTwb35R6TADBnIIVaOlk2NCE_4H0D5ALycs1Mcnk_xqD5x7fJVO-Zd4WT1JuOjWjCae_acDidXvzQpIS81E/s640/memleak-NO.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8wVws4QN_kmXo5zFB2Wr_eKGXCwbHFp30LOjd7Mp_4Tqc2Lk_Tw589UQAgYHjfA_292GJDADMO1N4rxyiIHfskQI3_WFp8fo09bo_-6ViU0Rslvl5oRJySw89sq1Xw-GLfY3y_M9Nh_4/s1600/memleak-YES.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="165" data-original-width="1043" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8wVws4QN_kmXo5zFB2Wr_eKGXCwbHFp30LOjd7Mp_4Tqc2Lk_Tw589UQAgYHjfA_292GJDADMO1N4rxyiIHfskQI3_WFp8fo09bo_-6ViU0Rslvl5oRJySw89sq1Xw-GLfY3y_M9Nh_4/s640/memleak-YES.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div class="myaux">
Note that using this method to troubleshoot parts of your VBA macro won't always work because many functions are prohibited inside VBA functions (they abort immediately, basically ensuring that VBA functions have no <a href="https://en.wikipedia.org/wiki/Side_effect_(computer_science)" target="_blank">side effects</a>). But it is very easy to modify the code above to be callable as a procedure from within a VBA macro.</div>
<div>
<br /></div>
<div>
<br /></div>
</div>
Sergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-87911442391161715132018-08-07T16:26:00.002-04:002018-08-07T16:46:36.900-04:00Laurel / Yanny Hands-On<div dir="ltr" style="text-align: left;" trbidi="on">
Ever since the <a href="https://en.wikipedia.org/wiki/Yanny_or_Laurel">Laurel/Yanny auditory illusion</a> went viral, I had a suspicion that we are dealing with a bifurcation-type illusion similar to the <a href="https://en.wikipedia.org/wiki/Figure%E2%80%93ground_(perception)">"figure or ground"</a> type:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://upload.wikimedia.org/wikipedia/commons/thumb/b/b5/Rubin2.jpg/250px-Rubin2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="190" data-original-width="250" src="https://upload.wikimedia.org/wikipedia/commons/thumb/b/b5/Rubin2.jpg/250px-Rubin2.jpg" /></a></div>
<br />
That guess seemed correct with the publishing of a NY Times article where you can "move a slider" to augment the clip in either direction, causing either "Laurel" or "Yanny" to be more pronounced.<br />
<br />
I admit that this article is quite revealing and educating (it taught me how I can tweak my brain into hearing one or the other). But moving a slider still seems rather artificial. For all I know they could have been cheating, actually having two separate recordings and the slider mixing them in different proportions.<br />
<br />
Can we prove this wasn't cheating? Hands on?<br />
<br />
Yes we can.<br />
<br />
Some snooping around reveals that here is a working version of the original recording:<br />
<a href="https://ia802800.us.archive.org/28/items/YannyVsLaurelVideoWhichNameDoYouHear-Audio/Yanny%20vs%20Laurel%20video%20which%20name%20do%20you%20hear%20%E2%80%93%20audio.mp3"><span style="font-size: xx-small;">https://ia802800.us.archive.org/28/items/YannyVsLaurelVideoWhichNameDoYouHear-Audio/Yanny%20vs%20Laurel%20video%20which%20name%20do%20you%20hear%20%E2%80%93%20audio.mp3</span></a><br />
<br />
Let's go to <a href="http://sandbox.open.wolframcloud.com/" target="_blank">Wolfram Cloud Computing</a> and run this simple program:<br />
<br />
<pre class="brush:c">url="(the url above)"
audio=AudioTrim[Audio[url],{0,4}]
CloudExport[AudioPitchShift[audio,1.1],"wav"] (*Laurel*)
CloudExport[AudioPitchShift[audio,0.85],"wav"] (*Yanny*)
</pre>
<br />
We see that at the heart of it is <span class="mycode">AudioPitchShift</span> which simply shifts the pitch of the recording by the desired amount. Looks like lower frequencies emphasize "Yanny" while higher frequencies bring out "Laurel", in agreement with the above mentioned Wikipedia article.
<br />
<br />
My conjecture, based on the observation that "Laurel" is lower-frequency than "Yanny", is that we tend to hear whatever is closer to the maximum frequency sensitivity of our ears, so whoever initially hears "Yanny" likely has an ear for higher-pitched tunes than the "Laurel" guy.
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRjWFrWABTkUnApxJf8ZEcRjsMjBg7m3zajFyuCf0ebxPoQfnIGHaw7iXf94oxEnKG2vSTuLL-6i6G8c0PfGsP6zZY9dv8f1h0_7rpfqSlB9mO0TkzIlPaYG54V06cgB_pjcvvZ6MjpRA/s1600/Untitled+document.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="394" data-original-width="615" height="205" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRjWFrWABTkUnApxJf8ZEcRjsMjBg7m3zajFyuCf0ebxPoQfnIGHaw7iXf94oxEnKG2vSTuLL-6i6G8c0PfGsP6zZY9dv8f1h0_7rpfqSlB9mO0TkzIlPaYG54V06cgB_pjcvvZ6MjpRA/s320/Untitled+document.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<br />
<div class="myaux">
It could have been a bit easier if Wolfram Cloud could actually play audio from within the interface without resorting to <span class="mycode">CloudExport</span>. Still, it opens nearly endless possibilities for experiments. Enjoy!</div>
<br /></div>
Sergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-8132796698391102572018-08-06T15:24:00.001-04:002018-08-06T15:27:44.733-04:00Old Phone/Tablet as an Info Board: Table of Contents<div dir="ltr" style="text-align: left;" trbidi="on">
Hi folks, now that this long overdue series of posts is complete, here's a table of contents for the ease of the reading.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyamVimcgaV5-vegUeCzM99BVMeqPx8j6RxD2NbXD5x4cbmjB-tAFrfLkYyztyG_i7jRqIhvhlCCsG8q9NU2M1jtFV3ktBH-P-gyVPwD64ZZjjlVqwY9a6ESnSjhPWRdRrwFcFWJM2wMU/s1600/canva-photo-editor.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="219" data-original-width="404" height="173" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyamVimcgaV5-vegUeCzM99BVMeqPx8j6RxD2NbXD5x4cbmjB-tAFrfLkYyztyG_i7jRqIhvhlCCsG8q9NU2M1jtFV3ktBH-P-gyVPwD64ZZjjlVqwY9a6ESnSjhPWRdRrwFcFWJM2wMU/s320/canva-photo-editor.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<br />
<ul style="text-align: left;">
<li><a href="http://curiousercode.blogspot.com/2018/01/old-phonetablet-as-info-board-intro.html">Intro: Backstory and Basics</a></li>
<li><a href="http://curiousercode.blogspot.com/2018/01/old-phonetablet-as-info-board-part-1.html">Part 1: IFRAME's and their limitations</a></li>
<li><a href="http://curiousercode.blogspot.com/2018/03/old-phonetablet-as-info-board-part-2.html">Part 2: Direct API queries</a></li>
<li><a href="http://curiousercode.blogspot.com/2018/06/old-phonetablet-as-info-board-part-3.html">Part 3: IFRAME with PHP capture-and-restyle</a></li>
<li><a href="http://curiousercode.blogspot.com/2018/08/old-phonetablet-as-info-board-part-4.html">Part 4: IFRAME with PHP capture-and-rearrange</a></li>
<li><a href="http://curiousercode.blogspot.com/2018/08/old-phonetablet-as-info-board-part-5.html">Part 5: PHP capture-and-process</a></li>
<li><a href="http://curiousercode.blogspot.com/2018/08/old-phonetablet-as-info-board-final.html">Final Part: Flow Control and Asynchronous Dynamic Data Refresh</a></li>
</ul>
<div>
Enjoy!</div>
<br />
<div class="myaux"><b>TL/DR</b>: This is a series of (moderately boring) posts about how to turn your old and outdated tablet or smartphone into an information board you can use at home. Uses range from purely practical to purely aesthetic.</div></div>
Sergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-39498132022779866962018-08-06T12:40:00.000-04:002018-08-06T15:14:56.974-04:00Old Phone/Tablet as an Info Board Final Part: Flow Control and Asynchronous Dynamic Data Refresh<div dir="ltr" style="text-align: left;" trbidi="on">
This is the final post about how to make an information board out of your old tablet or smartphone.<br />
<br />
Let's assume you have all your elements that deliver your information as you want it, beautifully designed and tested, again using my board as an example:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhx-In-ubR9dnUqjfkPQdFOstT675Nq_0WcC1tawUoX7ymbN1dOFRjhpQVOKKSg5ADGgECn14G2cpfNfMEbr1I9JgP820dCn0Y5J3XTf0y-xhUDumU3zmmfVSZxV3DelolpL1VVXjAoYoU/s1600/20180114_163623.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhx-In-ubR9dnUqjfkPQdFOstT675Nq_0WcC1tawUoX7ymbN1dOFRjhpQVOKKSg5ADGgECn14G2cpfNfMEbr1I9JgP820dCn0Y5J3XTf0y-xhUDumU3zmmfVSZxV3DelolpL1VVXjAoYoU/s400/20180114_163623.jpg" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<br />
OK, the information is up to date when the page has loaded. Now, there comes a question: <i>how to <b>keep</b> all this information reasonably up to date</i>, ready for you at any moment in case you need it? On-demand refresh of any kind is not an option - an info board you'd have to stand in front of, waiting, is a lousy info board.<br />
<br />
It would seem easy at first - "just do a <span class="mycode">META REFRESH</span>", - but there are several pitfalls to think through:<br />
<br />
<ol style="text-align: left;">
<li>It is obvious that you need to refresh all the elements with some frequency. But <i>how to choose this frequency</i>? Too seldom, your info is out of date half the time. Too frequently, and your board will spend more time refreshing than actually showing the info, and you risk being kicked out by your data sources for DDOS'ing them. </li>
<li>Further elaborating on the idea, it is clear that some elements need a more frequent refresh than some others: you are quite OK with a weather forecast obtained half an hour ago, but train departures from half an hour ago are totally useless to you. </li>
<li>Furthermore, the refresh frequency should be time dependent. You don't need frequent traffic updates in the middle of the night, but you <i>do</i> need them during commute hours.</li>
<li>What to do if an element fails to refresh, or gets stuck? Should we perhaps retry sooner than its normal refresh cycle? Then again, what would be the reasonable timeout? Set it too short and you'll mistake normal wait times for "getting stuck".</li>
</ol>
<div>
What it all means that you will need to refresh <i>asynchronously</i> (i.e. some elements but not others), and <i>dynamically</i> (using time intervals that are element and time dependent), and you'll have to have a mechanism to check if the refresh was actually successful or needs to be retried. Also note that some elements (notably those that use jQuery on a hidden <span class="mycode">iframe</span>) will need a two-step refresh: some method to reload the <span class="mycode">iframe</span>, and another to process its contents <i>once it has loaded and rendered in DOM</i> (but no sooner).</div>
<br />
<br />
<div>
<br /></div>
<div>
So, let us begin by introducing a "refresh handler" like so:</div>
<br />
<pre class="brush:js">var dummydate = new Date();
var rExample = {freq: 30, // refresh every 30 minutes by default
prime: [{freq: 5, start:{h:7, m:30}, end:{h:8, m:30}}, // between 7:30 and 8:30 refresh every 5 minutes
{freq: 15, start:{h:15, m:30}, end:{h:20, m:30}}], // between 15:30 and 20:30 refresh every 15 minutes
last: dummydate, next: dummydate, fresh: true, overdue: 0, //internal variables
lag: 5, retry: 45, //internal constants
handleRefresh: function(){...}, // refresh function
handleReady: function(){...}, // "is ready" function
handlePost: function(){...} // post-refresh function for 2-stage refresh
};
</pre>
<br />
<br />
<div>
We see that this handler is a self-contained object that handles all refresh code for "something on the board". Then, given an array of these handlers, we define an event poller to be called periodically:</div>
<br />
<pre class="brush:js">const scale = 1; //seconds per minute: 1 for debugging; 60 for actual use
var REFRESH = [rWeather, rTrain1, rTrain2, rBus, rHourly, rMap, rTravel];
function pollEvent()
{
var now= new Date();
// add force refresh @ 2AM, ONCE
for (var i=0, len=REFRESH.length; i<len; i++)
{
var handle = REFRESH[i];
if(now > handle.next) {
try{
if (handle.handleReady()) {
handle.overdue=0;
var behind = parseInt((now.getTime() - handle.last.getTime())/1000/scale);
$("#ref"+(i+1)).text(behind);$("#ref"+(i+1)).css("color","green");
}
else { console.log(i+" not ready")
if (handle.overdue > 5) {console.log("***FORCE REFRESH***"); /*location.reload();*/};
handle.overdue++;
$("#ref"+(i+1)).text("x");$("#ref"+(i+1)).css("color","darkred");
};
console.log("refresh "+i);
handle.handleRefresh(); handle.fresh = true;
postprocess(handle);
handle.last = now;
handle.next = new Date(now.getTime() + 1000*((handle.overdue==0)?handle.retry:scale*timespan(handle,now)));
}
catch(ignore) {handle.overdue++; $("#ref"+(i+1)).text("X");$("#ref"+(i+1)).css("color","red"); handle.next = new Date(now.getTime() + 1000*handle.retry);}
}
}
setTimeout(pollEvent,100*scale); // call again, every 6 seconds in production
};
</pre>
<br />
<br />
<div>
Here <span class="mycode">timespan()</span> defines the refresh frequency for a given handler at a given time:</div>
<div>
<br /></div>
<br />
<pre class="brush:js"> function timespan(handle,stamp)
{
var ts = handle.freq;
for (var j=0; j<handle.prime.length; j++)
{
var thisprime = handle.prime[j];
if (stamp.getHours()>= thisprime.start.h && stamp.getHours()<= thisprime.end.h
&& ( stamp.getHours()>thisprime.start.h ||
(stamp.getHours() == thisprime.start.h && stamp.getMinutes()>= thisprime.start.m))
&& ( stamp.getHours()<thisprime.end.h ||
(stamp.getHours() == thisprime.end.h && stamp.getMinutes()<= thisprime.end.m))
) {ts = thisprime.freq;}
}
return ts;
}
</pre>
<br />
<br />
<div>
Now define another function to ensure two-step refresh happens as fast as possible:</div>
<div>
<br /></div>
<br />
<pre class="brush:js">function postprocess(handle){
try{
if(handle.handleReady) {console.log("++"); handle.handlePost();}
else {console.log("--");setTimeout(function(){postprocess(handle);}, 1000*((handle.fresh)?1:handle.lag)); handle.fresh=false; };
}
catch(ignore) {}
}
</pre>
<br />
<br />
<div>
It only remains to define some auxiliary routines</div>
<div>
<br /></div>
<br />
<pre class="brush:js">function frameload(frameid){$(frameid).attr("src",$(frameid).attr("src"));}
function nop(){return true;}
function clock(){
var now = new Date(); var h=now.getHours(); var m=now.getMinutes(); var s = now.getSeconds();
$("#nowclock").text(((h>=10)?h:("0"+h)) + ":" + ((m>=10)?m:("0"+m)) + ":" + ((s>=10)?s:("0"+s)));}
</pre>
<br />
<br />
<div>
and add an initial invocation upon document's DOM ready:</div>
<div>
<br /></div>
<br />
<pre class="brush:js">$( document ).ready(function()
{
var now = new Date(); var h=now.getHours(); var m=now.getMinutes();
$("#loadclock").text(((h>=10)?h:("0"+h)) + ":" + ((m>=10)?m:("0"+m)));
setInterval(clock, 1000);
setTimeout(pollEvent,3000);
});
</pre>
<br />
<br />
<br />
That's all. For reference here are the refresh handlers for all the elements:<br />
<br />
<br />
<pre class="brush:js"> var rWeather = {freq: 30,
prime: [{freq: 15, start:{h:7, m:30}, end:{h:8, m:30}},
{freq: 15, start:{h:15, m:30}, end:{h:20, m:30}}],
last: dummydate, next: dummydate, fresh: true, overdue: 0, lag: 5, retry: 45,
handleRefresh: function(){f(document, 'script', 'plmxbtn');},
handleReady: function(){return $("#weather").find(".city").length>0;},
handlePost:nop };
var rTrain1 = {freq: 120,
prime: [{freq: 10, start:{h:5, m:30}, end:{h:9, m:0}},
{freq: 2, start:{h:7, m:45}, end:{h:8, m:20}}],
last: dummydate, next: dummydate, fresh: true, overdue: 0, lag: 1, retry: 10,
handleRefresh: function(){frameload("#go1");},
handleReady: function(){return $("iframe#go1").contents().find(".currentDateMain").length>0;},
handlePost:trainhide };
var rTrain2 = {freq: 20,
prime: [{freq: 5, start:{h:7, m:30}, end:{h:8, m:30}}],
last: dummydate, next: dummydate, fresh: true, overdue: 0, lag: 1, retry: 10,
handleRefresh: function(){frameload("#go2");},
handleReady: function(){return $("iframe#go2").contents().find(".currentDateMain").length>0;},
handlePost:trainhide };
var rBus = {freq: 20,
prime: [{freq: 5, start:{h:6, m:15}, end:{h:9, m:0}},
{freq: 1, start:{h:7, m:45}, end:{h:8, m:15}}],
last: dummydate, next: dummydate, fresh: true, overdue: 0, lag: 1, retry: 10,
handleRefresh: function(){frameload("#miway");},
handleReady: function(){return $("iframe#miway").contents().find("#bustoken").length>0;},
handlePost:nop };
var rHourly = {freq: 60,
prime: [{freq: 15, start:{h:7, m:30}, end:{h:8, m:30}},
{freq: 15, start:{h:15, m:30}, end:{h:20, m:30}}],
last: dummydate, next: dummydate, fresh: true, overdue: 0, lag: 5, retry: 45,
handleRefresh: function(){frameload("#forecast");},
handleReady: function(){return $("iframe#forecast").contents().find("table.wxo-media")
.find("[headers='header1']").length>0;},
handlePost:forecast }; // *** add clear if exists
var rMap = {freq: 30,
prime: [{freq: 5, start:{h:7, m:30}, end:{h:8, m:30}},
{freq: 5, start:{h:15, m:30}, end:{h:18, m:00}}],
last: dummydate, next: dummydate, fresh: true, overdue: 0, lag: 10, retry: 45,
handleRefresh: refreshMap,
handleReady: function(){return true;},
handlePost:nop }; // *** add clear if exists?
var rTravel = {freq: 20,
prime: [{freq: 5, start:{h:7, m:30}, end:{h:8, m:30}},
{freq: 5, start:{h:15, m:30}, end:{h:18, m:30}},
{freq: 1, start:{h:7, m:50}, end:{h:8, m:10}},
{freq: 1, start:{h:17, m:00}, end:{h:17, m:30}}],
last: dummydate, next: dummydate, fresh: true, overdue: 0, lag: 5, retry: 45,
handleRefresh: navigate,
handleReady: function(){return ( parseInt($("#result1").text()) > 0
&& parseInt($("#result2").text()) > 0);},
handlePost:nop };
</pre>
<br />
<br />
<br />
<br />
<div class="myaux">
The code also includes forced refresh once daily at approximately 2:00 AM. This is done to prevent an occasional memory leak to screw our browser (we'll be running this 24/7 for months on end, remember?). I am leaving this part as an exercise for the reader - the flow chart is basically, "if the current time is between 2:00 and 3:00, and if the startup day is not the same as the current day, do <span class="mycode">location.reload()</span>".<br />
<br />
Still, I can see that the Playbook browser (or to be exact, an app called <a href="http://www.blackberryrc.com/blackberry-playbook/apps/2011/0709/BackLight-Override-v103.html" target="_blank">Backlight Override</a>, which is just a browser wrapper that additionally prevents the backlight from ever going off) does crash - once every 3-4 weeks. Well, for me, this is a fairly acceptable "mean time between failures", even though anything more frequent than once a week would already border on annoying.<br />
<br /></div>
<br /></div>
Sergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-41334898194116758102018-08-02T14:32:00.000-04:002018-08-02T14:32:11.003-04:00 Old Phone/Tablet as an Info Board Part 5: PHP capture-and-process<div dir="ltr" style="text-align: left;" trbidi="on">
Finally, I describe one more method of putting information on the info board, suitable when there is a very limited amount of information you need from the target page, and when your output will have very little, if anything, to do with the target page layout. It is also the only possible method when the server outputs data in any format other than HTML.<br />
<br />
The example is <a href="http://www4.mississauga.ca/PlanATrip/">our local bus company</a>, where I would like to get the bus departure info for a certain stop. In the browser, the result looks like this:
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgx1oBnAiXj0gd-9OQAHlwOMcXlmejtEYJT5sht8Cnou7Tjsvbcq4yW5ts8ci3V8FOJAE08Eb1N4JnI-k3ktFbjaIyTOD1gMIrNEnn17AFNNYGt69DPp-FLL4V5U1LjT0be35gJ0l-CCLQ/s1600/miway.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="555" data-original-width="545" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgx1oBnAiXj0gd-9OQAHlwOMcXlmejtEYJT5sht8Cnou7Tjsvbcq4yW5ts8ci3V8FOJAE08Eb1N4JnI-k3ktFbjaIyTOD1gMIrNEnn17AFNNYGt69DPp-FLL4V5U1LjT0be35gJ0l-CCLQ/s320/miway.png" width="313" /></a></div>
<br /><br />
Now the biggest difficulty about using the previously described methods is the lack of a clearly defined URL associated with the results page - the results are dynamically generated by the parent page's javascript. After some sniffing around, it was possible to extract a working request link in the format
<br />
<div class="mycode">http://www4.mississauga.ca/PlanATrip/NextPassingTimes/RequestNextPassingTimes ?suggestionInputIdentifier=(stop_number) &suggestionInputType=Stop &stopInputIdentifier= &mustBeAccessible=false</div>
<br />
which brings up the result in this JSON-like format:
<br /><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiqK7I3Q0MoTkkbAY2pHMp4LuDo8AX47yEfa2Y4THbp5RrfJ6e_9Vf6i1RHmuhaZu1sEkvdsyT1aYHKG88CjOa4vn31bo82uEt5XCx8pe1HjpM6Z1tjAl1ziqMS5-uQ27oHOjj7hJW8P84/s1600/miway-raw.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="404" data-original-width="861" height="297" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiqK7I3Q0MoTkkbAY2pHMp4LuDo8AX47yEfa2Y4THbp5RrfJ6e_9Vf6i1RHmuhaZu1sEkvdsyT1aYHKG88CjOa4vn31bo82uEt5XCx8pe1HjpM6Z1tjAl1ziqMS5-uQ27oHOjj7hJW8P84/s640/miway-raw.png" width="640" /></a></div>
<br /><br />
Well, because the departure information I am after is somewhere in this JSON, I decided to parse it entirely in PHP using <span class="mycode"><a href="http://php.net/manual/en/function.preg-match-all.php">preg_match_all</a></span>, extracting route number and arrival time for the next upcoming bus, as well as three more after it, like so:<br />
<br />
<pre class="brush:php"><?php
date_default_timezone_set('EST');
$opts = array('http'=>array('header' => "User-Agent:MyAgent/1.0\r\n"));
$context = stream_context_create($opts);
$url = "http://www4.mississauga.ca/PlanATrip/NextPassingTimes/RequestNextPassingTimes?suggestionInputIdentifier=1127&suggestionInputType=Stop&stopInputIdentifier=&mustBeAccessible=false";
$timestamp = date("H:i");
$code = file_get_contents($url,false,$context);
$code = strstr($code, 'NextPassingTimesList',false);
$code = strstr($code, 'NextPassingTimesLegend',true);
// now code contains only trip data
$rid = preg_match_all("/data-route-key=([^=]+)data-trip-key/",$code,$matches1);
$tid = preg_match_all("/NextPassingTimesTime ([^N]+)u003c\/div/",$code,$matches2);
for($i=0;$i<$rid;$i++)
{
$route = str_replace("\\","",$matches1[1][$i]); $route = str_replace("\"","",$route); $route = str_replace("r","",$route);$route = str_replace("n","",$route);
$routes[$i] = str_replace("~~East"," E",$route);
$time = $matches2[1][$i];
$id1=stripos($time,"u003e");
$time = substr($time,$id1+5,strlen($time)-($id1+5)-1);
$sid = stripos($time,"u003c");
if ($sid!==false) {$time = substr($time, 0, $sid-1);}
$times[$i] = $time;
}
$color = (strpos($times[0],"min")===false)?"rgb(255,128,128)":"red";
echo '<html><head></head><body>';
echo '<div style="color:rgb(255,128,128); font-family: Arial, Helvetica, sans-serif; font-size: 20px; position: relative; top: -5px;">';
echo '<span style="color:gray;">', $timestamp, '&nbsp;</span>';
echo '<span id="bustoken" style="background: ',$color,'; color:white;">', " ".$routes[0], '</span>', '<span style="font-weight: bold; color:',$color,'">', "&nbsp;", $times[0], '</span>';
if ($rid > 1 && $tid > 1)
{
echo "<br/>";
echo '<span style="color:rgb(255,235,235);">', $timestamp, '&nbsp;</span>';
for ($j=1; $j<(($rid>4)?4:$rid); $j++)
{
echo '<span style="background: rgb(255,128,128); color:white; font-size: 14px;">', " ".$routes[$j], '</span>', '<span style="font-weight: bold; color: rgb(255,128,128);font-size: 14px;">', "&nbsp;", $times[$j], '</span> &nbsp;';
};
}
echo '</div></body></html>';
?>
</pre>
<br /><br />
Putting it into our standard PHP iframe like this:
<pre class="brush: html">
<div id="bus" class="basic info" style="position: absolute; left: 150px; bottom: 270px; width: 415px; height: 50px; background:rgb(255,235,235);">
<iframe id="miway" scrolling="no" src="bus.php" style="height: 100px; width: 415px; border: none;"> </iframe>
</div>
</pre>
<br/> <br/>
The result looks like:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9xLt4E5K-svqeXQ8JRqDzt8-zJu4G4Bgdq0HlBe2qOKmEka8NyiQpXIIXikQ92RavX4mNizGyOOJzQUaGkKLqFSHk5-RJm2ck11r-PxyqZwLufsN8IgUopw8FZAN3ywevVOxFCP5Wo4I/s1600/busboard.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="217" data-original-width="1321" height="65" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9xLt4E5K-svqeXQ8JRqDzt8-zJu4G4Bgdq0HlBe2qOKmEka8NyiQpXIIXikQ92RavX4mNizGyOOJzQUaGkKLqFSHk5-RJm2ck11r-PxyqZwLufsN8IgUopw8FZAN3ywevVOxFCP5Wo4I/s400/busboard.png" width="400" /></a></div>
<br />
<br />
<div class="myaux">
Note the following tricks:
<ul>
<li>Whenever the real-time information is available ("the bus is leaving in XX minutes"), I am accenting the color to highlight it.</li>
<li>I am also outputting the time of the query, so that the "XX minutes" format remains meaningful without having to query every single minute.</li>
<li>The line with <span class="mycode">date_default_timezone_set</span> serves to ensure correct DST for <span class="mycode">$timestamp</span>.</li>
<li>As a future development, I plan to supplement the "XX minutes" format with actual projected time, replacing it with "HH:MM (XX min)" and retaining accentuation.</li>
<li>As another exercise, I plan to change color accentuation depending on the remaining time - highlighting buses that are still reachable given the walking/running distance to the stop, and dimming the buses that aren't.</li>
</ul>
</div>
<br />
I admit that this method, as implemented, is quick-and-dirty and extremely brute-force, and is rather vulnerable to the underlying data format changes. A much more robust way would be to have the PHP parse the JSON "the proper way" and either process it using <a href="http://php.net/manual/en/book.json.php">dedicated JSON functions</a>, or simply <span class="mycode">echo</span> the HTML result in its entirety, using jQuery to interact with it. However, my approach has worked surprisingly well for over 6 months already - and "if it ain't broken, won't fix it".
</div>
Sergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-7378954849381684042018-08-01T16:15:00.001-04:002018-08-01T16:21:52.767-04:00 Old Phone/Tablet as an Info Board Part 4: IFRAME with PHP capture-and-rearrange<div dir="ltr" style="text-align: left;" trbidi="on">
As we can see from the <a href="https://curiousercode.blogspot.com/2018/06/old-phonetablet-as-info-board-part-3.html">previous post</a>, the capture-and-restyle method is ill-suited for larger target pages from which you only need a limited portion of data, and/or when you want to rearrange that data significantly. The reason is that it will take an enormous amount of analysis and restyling to get things look the way you want - on par with the effort needed to design a web site yourself. <br />
<br />
Here I describe another, complementary approach, which I dub <b>capture and rearrange</b>, that you can use in exactly the opposite scenario:<br />
<br />
<ul style="text-align: left;">
<li>your target page is not very lightweight,</li>
<li>you only need a small portion of the target's contents,</li>
<li>you need to rearrange the layout significantly. </li>
</ul>
<br/><br/>
<div>
As an example, we will use <a href="https://weather.gc.ca/canada_e.html">Environment Canada</a>'s hourly forecast page to display a limited portion of the page (the hourly forecast) in a totally different format - horizontal rather than vertical layout, a much more condensed presentation, and adding visual aids and highlighting according to the weather conditions. </div>
<br/>
<div>
Or in an example of a picture that's worth a thousand words, we would like to make this<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxHYSHZscWFI9DL_aMUudTPn-NbAZJ7fI-EPgBt8wGr9_UUx5EFe9HDI34NcvZo77kmyCVqAFrDpaXbhT360cYF-8eb8n0XGl4OIsfxukJyXTqHlXsPxUQTQRmmErBLE9_10y6QDiNv-w/s1600/envcan-after.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="278" data-original-width="1600" height="108" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxHYSHZscWFI9DL_aMUudTPn-NbAZJ7fI-EPgBt8wGr9_UUx5EFe9HDI34NcvZo77kmyCVqAFrDpaXbhT360cYF-8eb8n0XGl4OIsfxukJyXTqHlXsPxUQTQRmmErBLE9_10y6QDiNv-w/s640/envcan-after.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
from this (never mind the difference in the actual content; you get the idea)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1obZK_eiLae1CH2UedZ6NooEPSoVVUMGd_8w5OkFDwjNjJIfjGEXCjawHEsaIdxpqQVpd_JxyDdJXtMCfFhysBJBCdsFgpn3DILKI6Z0VrDqwYxPQHr0WOyexyPN8Kzp_-VFFiF-rSDk/s1600/envcan-before.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="615" data-original-width="759" height="259" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1obZK_eiLae1CH2UedZ6NooEPSoVVUMGd_8w5OkFDwjNjJIfjGEXCjawHEsaIdxpqQVpd_JxyDdJXtMCfFhysBJBCdsFgpn3DILKI6Z0VrDqwYxPQHr0WOyexyPN8Kzp_-VFFiF-rSDk/s320/envcan-before.png" width="320" /></a></div>
<br />
<br /></div>
<div>
The workflow is as follows:</div>
<div>
<ol style="text-align: left;">
<li>Capture the page via PHP in the previously described way, like so:<br />
<pre class="brush: php">
<?php
$opts = array('http'=>array('header' => "User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.59.10 (KHTML, like Gecko) Version/5.1.9 Safari/534.59.10\r\n"));
$context = stream_context_create($opts);
$code = file_get_contents("http://weather.gc.ca/forecast/hourly/on-24_metric_e.html",false,$context);
$code = str_replace("/weathericons/small/","http://weather.gc.ca/weathericons/small/",$code);
$code = str_replace("Medium","Med",$code);
echo $code;
?>
</pre>
<br /><br/>Note that we have redirected the images directly to the server to spare the effort of saving locally.
<br />
<div class="myaux">
However, in my particular case, I found out that the Playbook browser (unlike newer browsers) had trouble accessing the server version of the images. Well, if you have read the previous post, you know the workaround: just create a local copy of <span class="mycode">weathericons/small/</span> and do this in any bash compatible terminal (Mac/Linux/Cygwin) - <i>do not forget to set the execute attribute on the DiskStation</i>:
<pre class="brush:bash">
for $i in $(seq 0 9); do wget https://weather.gc.ca/weathericons/small/0$i.png; done
for $i in $(seq 10 99); do wget https://weather.gc.ca/weathericons/small/$i.png; done
</pre>
</div>
</li>
</ol>
</div>
<br /><br />
<li>Then define a placeholder interface for our forecast, displaying our PHP captured page in a hidden <span class="mycode">iframe</span>:<br/>
<pre class="brush: html">
<div id="hourly" class="basic info" style="position: absolute; left: 10px; bottom: 10px; width: 990px; height: 160px; background: white;">
<iframe id="forecast" style="visibility: hidden; height: 0px !important;" src="hourly.php"> </iframe>
<table > <tbody>
<tr id="rowTime" class="element">
</tr>
<tr id="rowTemp" class="element" style="font-weight: bold; font-size: 16px;">
</tr>
<tr id="rowIcon" class="element">
</tr>
<tr id="rowText" class="element" style="font-size: 7px;">
</tr>
<tr id="rowRain" class="element">
</tr>
<tr id="rowWind" class="element">
</tr>
<tr id="rowChill" class="element">
</tr>
</tbody></table>
</div>
</pre>
</li>
<li>Inspect the HTML of our source page to uniquely identify elements you want to use in your widget. This is the trickiest but the most creative part - I will detail what I did in my example, but it will be entirely dependent on the source website. As on many other occasions, the Inspect Element functionality in your browser is the tool of choice here. Fortunately, most websites nowadays are designed in such a way that all their elements are addressable by some combination of their IDs, class names and styles, which is what we need.</li>
<li>Once the frame is loaded, and once we know which elements to look for, we move those elements from the hidden iframe to our placeholder using jQuery's capabilities:
<br />
Like so:<br/>
<pre class="brush:js">
function forecast()
{
$(".element").empty();
var iFrameDOM = $("iframe#forecast").contents().find("table.wxo-media");
$("#rowTime").append(iFrameDOM.find("[headers='header1']"));
$("#rowTemp").append(iFrameDOM.find("[headers='header2']"));
$("#rowIcon").append(iFrameDOM.find("img.media-object")); $("#rowIcon").find("img.media-object").wrap("<td> </td>");
$("#rowText").append(iFrameDOM.find("div.media-body")); $("#rowText").find("div.media-body").wrap("<td> </td>");
$("#rowRain").append(iFrameDOM.find("[headers='header4']"));
$("#rowWind").append(iFrameDOM.find("[headers='header5']"));
$("#rowChill").append(iFrameDOM.find("[headers='header7']"));
$("#rowTemp").find("td").each(eachTemp);
$("#rowRain").find("td").each(eachRain);
$("#rowWind").find("td").each(eachWind);
$("#rowChill").find("td").each(eachChill);
}
</pre>
</li>
<br />
<br />
The final four lines in the above snippet introduce content-dependent formatting of the newly added elements - I want to be able to see if there's rain/wind/heat/freeze from across the room. For this, we write a few auxiliary functions.
<br />
<br />
Temperature gradient - this is just a linear interpolation between colors:<br />
<pre class="brush:js">
function gradient(deg)
{
var colors = [
[-15, 255,0,255],
[-10, 255,128,255],
[0, 128,128,255],
[12, 128,255,255],
[17, 150,255,128],
[25, 255,255,128],
[30, 255,128,0],
[35, 255,0,0]
];
// determine extrema
if (deg <= colors[0][0]) {return "rgb(" + colors[0][1] + "," + colors[0][2] + "," + colors[0][3] + ")";}
if (deg >= colors[7][0]) {return "rgb(" + colors[7][1] + "," + colors[7][2] + "," + colors[7][3] + ")";}
//otherwise, interpolate
for(i=1;i<=7;i++)
{
if(deg > colors[i-1][0] && deg <= colors[i][0])
{
var p = (deg-colors[i-1][0]) / (colors[i][0]-colors[i-1][0]);
return "rgb("
+ Math.round(colors[i][1]*p + colors[i-1][1]*(1.0-p)) + ","
+ Math.round(colors[i][2]*p + colors[i-1][2]*(1.0-p)) + ","
+ Math.round(colors[i][3]*p + colors[i-1][3]*(1.0-p)) + ")";
};
}
//error
return "rgb(255,255,0)";
}
function eachTemp(){$(this).css("background",gradient(parseInt($(this).text())));}
</pre>
<br />
Rain highlighting:<br />
<pre class="brush:js">
function eachRain(){
var str = $(this).text();
var result = "rgb(255,255,255)";var result2 = "rgb(0,0,0)";
if (str=="Low") {result = "rgb(225,255,255)";}
if (str=="Med") {result = "rgb(0,255,255)";}
if (str=="High") {result = "rgb(0,0,255)";result2 = "rgb(255,255,255)";}
$(this).css("background",result);
$(this).css("color",result2);
}
</pre>
<br />
Wind highlighting: <br />
<pre class="brush:js">
function eachWind(){
var str = $(this).text().trim().split(String.fromCharCode(160));
var result = "rgb(255,255,255)"; var speed = 0;
console.log(str);
try { speed = parseInt(str[1]);} catch(ignore){speed=0;}
if (speed >=20 ) {result = "rgb(255,255,128)";}
if (speed >=40 ) {result = "rgb(255,255,0)";}
if (speed >=60 ) {result = "rgb(255,0,0)";}
$(this).css("background",result);
}
</pre>
<br />
Wind chill highlighting - note that it will depend on <i>other</i> element's content (hence the need to look beyond <span class="mycode">$(this)</span> and therefore pass the <span class="mycode">index</span> parameter.)<br />
<pre class="brush:js">
function eachChill(index){
var str = $(this).text()
var result = "rgb(255,255,255)"; var chill = 0; var temp=0; var diff=0;
try { chill = parseInt(str); temp = parseInt($($("#rowTemp").find("td")[index]).text()); diff = chill-temp; }
catch(ignore){diff=0;}
if (diff<= -5) {result = "rgb(128,128,255)";}
if (diff >= 5 ) {result = "rgb(255,128,128)";}
$(this).css("background",result);
}
</pre>
<br />
<div>
The final result looks more or less like in the picture above. All that remains is to call <span class="mycode">forecast()</span> at some point after the hidden <span class="mycode">iframe</span> has finished loading. I'll describe this in more detail in the final post that has details on flow control (which seems simple, but ended up being rather sophisticated for the sake of usability).
</div>
<br/> <br/>
<div class="myaux">
Note that this design is not completely fool-proof. Table columns will change with depending on the content, some of the info may be clipped in some rare cases, and I have not yet extended the functionality to include both wind chill and humidex. All of these fixes may be considered exercises for the reader. After all, as a father of two with hardly any extended family support I had to become a firm believer in the <a href="https://en.wikipedia.org/wiki/Pareto_principle">Pareto principle</a>.
</div>
</div>
Sergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-52980560141150023342018-06-04T17:01:00.000-04:002018-06-05T14:41:20.753-04:00Old Phone/Tablet as an Info Board Part 3: IFRAME with PHP capture-and-restyle<div dir="ltr" style="text-align: left;" trbidi="on">
In an <a href="http://curiousercode.blogspot.com/2018/03/old-phonetablet-as-info-board-part-2.html">earlier post</a>, we discussed that using <span class="mycode">iframe</span> for an information widget on our screen is perhaps the most universal, but the least versatile because without the ability to interact with the content of the <span class="mycode">iframe</span>, you cannot customize anything at all about the way your information is presented. So if you happen to need to reorganize your information to fit your needs, bad luck.<br />
<br />
Luckily I have a <a href="https://www.synology.com/en-us">Synology DiskStation</a> to host my screen, and it can function as a PHP web server. So the idea becomes to fetch the target web page by the server and echo it back to the client, using code like this :<br />
<br />
<pre class="brush: php">
<?php
$opts = array('http'=>array('header' => "User-Agent:MyAgent/1.0\r\n"));
$context = stream_context_create($opts);
$code = file_get_contents("http://your_URL_goes_here",false,$context);
echo $code;
?>
</pre>
<br />
<span class="myinline">Here <span class="mycode">$opts</span> and <span class="mycode">$context</span> are needed to appear to the target host as a legitimate HTTP request so it won't respond with something like Error 403.</span> Now, if you put the above into a file called, e.g., <span class="mycode">frame.php</span>, you can then embed it into your interface using our old <span class="mycode"><iframe src="frame.php"...></span> tag.
<br />
<br />
As an immediate benefit, PHP <span class="mycode">get_file_contents</span> is not a browser, so it will retrieve only the basic HTML (hopefully containing the information you are after). This makes the resulting <span class="mycode">iframe</span> a whole lot, big time easier on the client browser!<br />
<br />
On the flipside, this also means that you will need to do the styling and formatting yourself.<br />
<br />
The first method that we cover is something I called "<b>capture-and-restyle</b>" or "CSS injection". In a nutshell, <i>the idea is to save a local copy of the target site resources (mainly, CSS styles and images), which you can then edit to suit your needs</i>. This method is best suitable if:<br />
<ul style="text-align: left;">
<li>Most of the target page contents will be used;</li>
<li>The target page, as formatted, looks more or less the way you want it to look on the board;</li>
<li>The target page is relatively simple - you don't want to sift through hundreds of styles manually.</li>
</ul>
<div>
As an example, we use the GO transit (suburban commuter trains in the Toronto area) mobile departure board page: <br/>
<a href="http://gotracker.ca/GoTracker/mobile/StationStatus/Service/01/Station/7">http://gotracker.ca/GoTracker/mobile/StationStatus/Service/01/Station/7</a>
<br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbzm7IVTs45L9y9NUbi4p8uvXK1yK1JbpMue5rMql7mg7f-FSRLcNUWEtRfOx598mOgfENrfHCM-2AwJMaSUqBuUfR_vxIriuQaf_POT3P0NKYGtI03QNWrsaONbZ2MisdkzJrNQrcewI/s1600/go-was.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbzm7IVTs45L9y9NUbi4p8uvXK1yK1JbpMue5rMql7mg7f-FSRLcNUWEtRfOx598mOgfENrfHCM-2AwJMaSUqBuUfR_vxIriuQaf_POT3P0NKYGtI03QNWrsaONbZ2MisdkzJrNQrcewI/s400/go-was.png" width="400" height="138" data-original-width="1494" data-original-height="517" /></a></div>
<br/>
It is already "kind of" optimized for viewing on small screens, so we will only need to make a few minor adjustments. So it makes sense to reuse as much of the original styling as we can.
</div>
<div>
<br /><br /></div>
<div>
The workflow is as follows:</div>
<div>
<ol style="text-align: left;">
<li>Save local copies of styles and images that you want to display in your page. (The simplest way of doing this is to save the complete webpage in Chrome and look for items). Upload these on the DiskStation in a subfolder (e.g. <span class="mycode">go/</span>) to the same folder where your <span class="mycode">index.html</span> resides:<br>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwA10hcR_K7gv0-m8ydpwcC8MRWH1RCI8rLp4Tjxz-bpTRJ4nqFe0P9mZGhwuQwTo2_qYr72tY1OYMIXZz-YSZSitWHTi3qWSQNXDSmsqx8XPLdERCdEiUT5VBKb9bPz5C9k-CTyVyI24/s1600/go-tree.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwA10hcR_K7gv0-m8ydpwcC8MRWH1RCI8rLp4Tjxz-bpTRJ4nqFe0P9mZGhwuQwTo2_qYr72tY1OYMIXZz-YSZSitWHTi3qWSQNXDSmsqx8XPLdERCdEiUT5VBKb9bPz5C9k-CTyVyI24/s400/go-tree.png" width="400" height="62" data-original-width="810" data-original-height="126" /></a></div>
<b>Important!</b> <i>Make sure that you grant Execute permissions to the images!</i> (To do this, right click the items in the File Station browser and choose Properties.)
</li>
<li>Add the following to the PHP script right before <span class="mycode">echo $code</span>, redirecting requests to styles and images to local copies:
<pre class="brush:php">
$code = str_replace('/GOTracker/mobile/', 'go/', $code);
$code = str_replace('/GoTracker/mobile/', 'go/', $code);
$code = str_replace('../../../../', 'go/', $code);
</pre>
At this point, your local PHP, entered in your local browser (e.g. <span class="mycode">http://192.168.0.xxx/frame.php</span>) should look (more or less) exactly as the target page typed in the same browser. If it does not, look for the missing elements / check permissions. Your web page inspector in Chrome or Safari is your friend here.</li>
<li>Edit the locally saved styles/images to augment the way the widget looks. Basically, in this case, we want to hide the elements that are irrelevant to us, such as the logos; shrink the sign of the direction banners (they are obvious to us); and adjust the size of the information bearing elements so that they format nicely in a small-sized widget.
I ended up adding a chunk of CSS code to the end of the stylesheet that loads last (<span class="mycode">GOGrid.css</span>), more or less like so:
<br />
<pre class="brush: css">
* {border: none !important; font-family: Arial, Helvetica, sans-serif !important; font-stretch: semi-condensed;}
#frontImg {visibility: hidden !important; height: 0px !important;}
.imageButtonLink {visibility: hidden !important; height: 0px !important;}
.sTbl {table-layout: fixed;}
.SecondTitle > td:nth-child(1), .SecondTitle > td:nth-child(3) {width:0px ;}
.SecondTitle > td:nth-child(2) { text-align: left !important; font-stretch: none; font-weight: bold;}
.headerTR {visibility: hidden !important; height: 0px !important; font-size:0px;}
.directionHeaderTH {font-size:6px ;}
.bottomDoubleRowTR * {font-weight: normal; font-size: 10px;}
.oddRowTR:nth-child(even) {background:rgb(225,255,225);}
.feedbackLink {visibility: hidden !important; height: 0px !important; font-size:0px;}
.currentDateMain {position: fixed !important; visibility: visible !important; top: 5px !important; right: 0px;}
#lblCurrentDateMain {color: rgb(128,150,128); text-shadow: none !important; }
#lblCurrentTimeMain {color: rgb(0,128,0); font-weight: bold; text-shadow: none !important; }
</pre>
<br />
<span class = "myinline">(Note the line with <span class="mycode">nth-child(even)</span>: this introduces the striped table style for ease of readability. Apparently this was originally in mind of the website programmers, since the class name, <span class="mycode">.oddRowTR</span>, kind of suggests that there should also be <span class="mycode">.evenRowTR</span> with different styling; however in practice all rows are of the <span class="mycode">.oddRowTR</span> class, so, well, we fixed this.:)</span></li>
<li>Done - it ended up looking like so:<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQP4NXtlJUeGfVHWnqobaH6QEtaLkDLPwzxQ6zz8JJMgc1jxqgKJ9D2EFNIkD-pDbrbcj8vKRzG-azZPe1JSlm-8eQoq2Qw4XP6lc_bjNS7eQSw_YTCnhypdo2aqZuPTDuVERH6Y5JllU/s1600/go-is.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQP4NXtlJUeGfVHWnqobaH6QEtaLkDLPwzxQ6zz8JJMgc1jxqgKJ9D2EFNIkD-pDbrbcj8vKRzG-azZPe1JSlm-8eQoq2Qw4XP6lc_bjNS7eQSw_YTCnhypdo2aqZuPTDuVERH6Y5JllU/s1600/go-is.png" data-original-width="216" data-original-height="215" /></a></div>
<br/>
</li>
</ol>
</div>
<br /><br />
So here is the final version of the PHP
<br />
<pre class="brush:php">
<?php
$opts = array('http'=>array('header' => "User-Agent:MyAgent/1.0\r\n"));
$context = stream_context_create($opts);
$line=$_GET["line"];
$stn=$_GET["station"];
$code = file_get_contents("http://gotracker.ca/GoTracker/mobile/StationStatus/Service/".$line."/Station/".$stn,false,$context);
$code = str_replace('/GOTracker/mobile/', 'go/', $code);
$code = str_replace('/GoTracker/mobile/', 'go/', $code);
$code = str_replace('../../../../', 'go/', $code);
$code = str_replace('Union Station', 'Union Stn', $code);
$code = str_replace('Clarkson GO', 'Clarkson', $code);
$code = str_replace('Erindale GO', 'Erindale', $code);
$code = str_replace('On Time', 'OK', $code);
echo $code;
?>
</pre>
<div class="myaux">Notice that I added some parametric functionality to reuse the same PHP for two stations as in the <a href="http://curiousercode.blogspot.com/2018/01/old-phonetablet-as-info-board-intro.html">original layout</a>.
The last four <span class="mycode">str_replace</span> are for cosmetic purposes - to make the individual departure lines fit on a single line as often as possible. There will still be occasional ugly misses, but (1) they can be corrected upon discovery, and (2) nothing is perfect, so why bother.
</div>
<br/>
and final HTML
<br />
<pre class="brush:html">
<div id="train1" class="basic" style="position: absolute; left: 150px; top: 10px; width: 200px; height: 195px; ">
<iframe id="go1" scrolling="no" src="go.php?line=21&station=554" style="height: 190px; width: 200px; border: none;"> </iframe>
</div>
<div id="train2" class="basic" style="position: absolute; left: 365px; top: 10px; width: 200px; height: 195px; ">
<iframe id="go2" scrolling="no" src="go.php?line=01&station=7" style="height: 190px; width: 200px; border: none;"> </iframe>
</div>
</pre>
<br />
As another big benefit, the source of the <span class="mycode">iframe</span> now has the same origin as your main page. This means that you are now fully in control of its contents, which can now be accessed and manipulated from the script. We will make extensive use of this feature in the next post where we describe another PHP+iframe combo to make our hourly weather forecast widget. Here, we limit the use of this feature to a simple example: dim the widget if it has no relevant information. Let us do it like this:
<br />
<pre class="brush:html">
<div id="mask1" class="basic" style="opacity: 0.75; visibility: hidden; position: absolute; left: 150px; top: 10px; width: 200px; height: 195px; "></div>
<div id="mask2" class="basic" style="opacity: 0.75; visibility: hidden; position: absolute; left: 365px; top: 10px; width: 200px; height: 195px; "></div>
</pre>
<pre class="brush:js">
function trainhide()
{
var dom = $("iframe#go1").contents().find(".oddRowTR");
$("#mask1").css("visibility",(dom.size()==0)?"visible":"hidden");
dom = $("iframe#go2").contents().find(".oddRowTR");
$("#mask2").css("visibility",(dom.size()==0)?"visible":"hidden");
}
</pre>
<br />
<div class="myaux">
Interlude: I apologize that it takes me so long to write up this series of posts; however the need to meticulously sort though all the necessary snippets and screenshots is taking more time than I previously thought. Bear with me - there are "only" 3 posts left. In the meantime, the info board continues to work - for nearly 6 months already.
</div>
</div>
Sergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-49710922951702825622018-03-16T20:00:00.000-04:002018-03-16T20:00:29.867-04:00Old Phone/Tablet as an Info Board Part 2: Direct API queries<div dir="ltr" style="text-align: left;" trbidi="on">
In the previous post, we discussed using iframe for an information widget on our info board, noting that it would be too heavy on your browser if your target web page is feature-rich. This means you can forget anything like a Google maps snippet.<br />
<br />
However, precisely with this in mind, many large websites have developed APIs that other websites (read: we) can use. Google does a great job providing its <a href="https://developers.google.com/maps/">maps API</a>, so incorporating local traffic can be done as simple as<br />
<br />
<pre class="brush: js; html-script: true">
<div id="map" class="basic" style="border: none; position: absolute; right: 10px; top: 10px; width: 415px; height: 350px;">
</div>
<script async defer
src="https://maps.googleapis.com/maps/api/js?key=(get_your_own_key)&callback=initMap">
</script>
<script>
function initMap() {
map = new google.maps.Map(document.getElementById('map'), {
zoom: 13,
center: {lat: 44.444, lng: -77.777} // Change these parameters to your location
});
var trafficLayer = new google.maps.TrafficLayer();
trafficLayer.setMap(map);
}
</script>
</pre>
<br /><br />
Obviously to make this work, you will need to <a href="https://developers.google.com/maps/documentation/javascript/get-api-key">get you own Google API key</a>, and change the map location (ans possibly, zoom) to where you would like your map to display. The easiest way to find it would be to open your Google Maps view and then look at the URL to find the values for <a href="https://en.wikipedia.org/wiki/Geographic_coordinate_system">latitude and longitude</a>.
<br /><br />
Now let us take a step further and provide a widget to monitor the duration in traffic of a given trip in real time. For added flavor, we will compute both the travel time in both directions, where the outbound trip is to start now and the return trip would start in the future.
<br /><br />
For this, we define the interface<br />
<br />
<pre class="brush: html">
<div id="travel" class="basic info" style="position: absolute; left: 150px; bottom: 190px; width: 415px; height: 60px; background:rgb(255,255,225);">
<span id="log1" style="font-size: 10px; color:rgb(200,200,200); position: relative; top: 0px;">Not loading</span><br/>
<span id="log2" style="font-size: 10px; color:rgb(200,200,200); position: relative; top: 2px;">Not loading</span><br/>
<span id="results" style="position: relative; top: 10px;"><span id="result1" style="font-weight: bold;">. . .</span> / <span id="result2" style="font-weight: bold;">. . .</span> <span id="resultkm">...</span></span>
</div>
</pre>
<br /><br />
and poll the Directions API like this:<br />
<br />
<pre class="brush: js">
function navigate(){
$("#log1").text("load...");$("#log2").text("load...");
var origin = "44.44444, -77.77777";
var destination = "43.33333,-78.88888";
//Change these to your origin and destination
var directionsService = new google.maps.DirectionsService();
// var directionsDisplay = new google.maps.DirectionsRenderer();
// directionsDisplay.setMap(map)
var request1 = {
origin: origin, destination: destination,
travelMode: google.maps.DirectionsTravelMode.DRIVING,
drivingOptions: { departureTime: new Date(Date.now()+0), trafficModel: 'bestguess'}
};
var request2 = {
origin: destination, destination: origin, // return trip
travelMode: google.maps.DirectionsTravelMode.DRIVING,
drivingOptions: { departureTime: new Date(Date.now()+1000*60*30), trafficModel: 'bestguess'}
// trip stars 30 minutes in the future
};
var fAlert = "rgb(200, 0, 200)";
var fWarn = "rgb(255, 0, 0)";
var bOK = "rgb(225,255,225)";
var bAlert = "rgb(255,255,200)";
var bWarn = "rgb(255,180,180)";
directionsService.route( request1, function( response, status ) {
if ( status === 'OK' ) {
// directionsDisplay.setDirections(response);
var point = response.routes[ 0 ].legs[ 0 ];
var d=new Date();
$( '#log1' ).html(d.toString());
$( '#result1' ).html(point.duration_in_traffic.text);
$( '#resultkm' ).html(' (' + point.distance.text + ')' );
if (point.duration_in_traffic.value > 16*60) {$("#result1").css("color",fAlert);};
if (point.duration_in_traffic.value > 25*60) {$("#result1").css("color",fWarn);};
$("#travel").css("background-color",bOK);
if ($("#result1").css("color")==fAlert) {$("#travel").css("background-color",bAlert);};
if ($("#result1").css("color")==fWarn || $("#result2").css("color")==fWarn ) {$("#travel").css("background-color",bWarn);};
}
} );
directionsService.route( request2, function( response, status ) {
if ( status === 'OK' ) {
var point = response.routes[ 0 ].legs[ 0 ];
var d=new Date();
$( '#log2' ).html(d.toString());
$( '#result2' ).html(point.duration_in_traffic.text);
if (point.duration_in_traffic.value > 16*60) {$("#result2").css("color",fAlert);};
if (point.duration_in_traffic.value > 25*60) {$("#result2").css("color",fWarn);};
}
} );
}
</pre>
<div class="myaux">
Note that as per Google's use policies, you will need to display the directions and route on a map if you use this function on a website that other people can see - just uncomment the corresponding <span class="mycode">DirectionsRenderer</span> code lines. <span class = "myinline">For your own use, and at your own risk, I believe you can forego this for the sake of readability if you know what you are doing. There is a caveat, though, in that you won't know the exact route that the time was calculated for.</span>
</div>
<br />
Here are the examples, showing three possible traffic situations:
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCX0X-VULOoz2C5dkC_6HSzUzmcr34EYK3Baeu7Mw0JigU0IP9NvYCztPhuDjIHgYsqbJHzAaiNOUmWzOteK1DMAQaH7t7VXmgCWzTo92GPk_6hNLxIkq0CfAUOnvmn8GdiawUVCrbvPc/s1600/map_comb.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCX0X-VULOoz2C5dkC_6HSzUzmcr34EYK3Baeu7Mw0JigU0IP9NvYCztPhuDjIHgYsqbJHzAaiNOUmWzOteK1DMAQaH7t7VXmgCWzTo92GPk_6hNLxIkq0CfAUOnvmn8GdiawUVCrbvPc/s400/map_comb.png" width="400" height="141" data-original-width="778" data-original-height="275" /></a></div>
As usual, your mileage may vary:<br />
<ul style="text-align: left;">
<li>You can implement more sophisticated logic of coloring and alerts - an obvious candidate would be analyzing the difference between <span class="mycode">.duration</span> and <span class="mycode">.duration_in_traffic</span></li>
<li>You can analyze the suggested best route and alert if it happens to be different from the default route (which you can determine by making a request when there is as little traffic as possible, i.e. in the middle of the night) - this will indicate a major congestion.</li>
<li>You can compare travel time along a predetermined route (by setting many waypoints in the most congested portion) and the chosen preferred route;</li>
<li class="myinline">You can even log / graph the travel time gathering your own statistics and storing it in an SQL database on the DiskStation. This way, should Google's API become unavailable, you will have a fallback method to determine travel times based on historical values. </li>
</ul>
<div class="myaux">
As far as the billing goes... Even if we poll the server every minute round the clock (which is totally overkill), we still won't exceed Google free usage quotas. And even if we did, once debugged and calibrated, this is such a useful service that you may seriously consider paying for it.</div>
</div>
Sergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-39958263175149500372018-01-18T17:30:00.000-05:002018-01-18T17:30:27.927-05:00Old Phone/Tablet as an Info Board Part 1: IFRAME's and their limitations<div dir="ltr" style="text-align: left;" trbidi="on">
This continues our series on making an info screen. Last time, we created a skeleton layout, so let us begin filling it with contents.<br />
<br />
As an example, let us try to display current and hourly weather using this webpage:<br />
<a href="https://www.theweathernetwork.com/ca/hourly-weather-forecast/ontario/mississauga" target="_blank">https://www.theweathernetwork.com/ca/hourly-weather-forecast/ontario/mississauga</a><br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieBECVGjapJpvmeqqmtjTVEFMhDw4zVKCQMb0tG4rVoeiW0Fg1l-PWGK-KUCk7QysJP8shDBMFpW42l0cqlS_2oSHPKDKE-3e5XwZ175lUPbj5s7YFfBoDA01ak1hN3iy0zkD5bnIf8Dc/s1600/weatherscreen1.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1015" data-original-width="975" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieBECVGjapJpvmeqqmtjTVEFMhDw4zVKCQMb0tG4rVoeiW0Fg1l-PWGK-KUCk7QysJP8shDBMFpW42l0cqlS_2oSHPKDKE-3e5XwZ175lUPbj5s7YFfBoDA01ak1hN3iy0zkD5bnIf8Dc/s400/weatherscreen1.png" width="383" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Highlighted are regions I'd like to put on the info board.</td></tr>
</tbody></table>
<br />
The most straightforward way of putting something from the third-party website into your own would be <a href="https://www.w3schools.com/tags/tag_iframe.asp" target="_blank">an <span class="mycode">IFRAME</span> tag</a>, of this global format:<br />
<div class="mycode">
<IFRAME scrolling="no" src="..." style="..."></IFRAME>
</div>
<br />
Now <i>usually</i> you would only need some portion of the website displayed on your screen. Unfortunately, if the contents of your <span class="mycode">IFRAME</span> is from the third-party website, you cannot interact with its content (with a very few exceptions) due to the commonly accepted <a href="https://en.wikipedia.org/wiki/Same-origin_policy" target="_blank">same-origin policy</a>. (Annoying as it is in our case, this limitation is what prevents a fair amount of malicious attacks.)<br />
<br />
However, the desired portion of the website can be extracted via re-positioning the iframe using this <a href="http://www.dimpost.com/2012/12/iframe-how-to-display-specific-part-of.html" target="_blank">negative margin trick</a>:<br />
<br />
<div class="mycode">
style="margin-left: -(XXX)px; margin-top: -(YYY)px;"
</div>
<br />
To zoom in or out on the corresponding website, the only way is to use <a href="https://www.w3schools.com/cssref/css3_pr_transform.asp" target="_blank">CSS transform property</a>, like so:<br />
<br />
<pre class="brush:html">
<div id="weather1" class="basic" style="position: relative; left: 5px; top: 10px; width: 500px; height: 180px;">
<iframe scrolling="no" src="https://www.theweathernetwork.com/ca/hourly-weather-forecast/ontario/mississauga"
style=" -webkit-transform: scale(0.72); -webkit-transform-origin: 0 0;
transform: scale(0.72); transform-origin: 0 0;
margin-left: -20px; margin-top: -200px;
border: 0px none; height: 8120px; width: 750px;">
</iframe></div>
<div id="weather2" class="basic" style="position: relative; left: 5px; top: 20px; width: 500px; height: 200px;">
<iframe scrolling="no" src="https://www.theweathernetwork.com/ca/hourly-weather-forecast/ontario/mississauga"
style=" -webkit-transform: scale(0.72); -webkit-transform-origin: 0 0;
transform: scale(0.72); transform-origin: 0 0;
margin-left: -30px; margin-top: -690px;
border: 0px none; height: 8120px; width: 750px;">
</iframe></div>
</pre>
(The <span class="mycode">-webkit-</span> prefix is needed for browsers like the PlayBook's (or Safari); some other browsers may need other prefixes.)
<br />
This gives us something like:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgl-DWOTXmAvKDrgivGF-j0lo4NOBkGlTSNoSYmM2Cdn-IPyAj2SIt7i9pkFVnYRye4PpKFQFsM51k_SkPEq6v8Y2BwaafZlccT0-V2KUAsoUWcQwR-tgdj2t-0pxsKzLXdmqTRGRQQFsE/s1600/weathersnip1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="380" data-original-width="468" height="259" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgl-DWOTXmAvKDrgivGF-j0lo4NOBkGlTSNoSYmM2Cdn-IPyAj2SIt7i9pkFVnYRye4PpKFQFsM51k_SkPEq6v8Y2BwaafZlccT0-V2KUAsoUWcQwR-tgdj2t-0pxsKzLXdmqTRGRQQFsE/s320/weathersnip1.png" width="320" /></a></div>
<br />
As you can see, this method is very easy to code and there is no need (and no possibility for that matter) to do any rearrangement of the displayed information. The target website takes care of that for you.<br />
<br />
<br />
Two major pitfalls (besides the above mentioned inability to interact with the iframe contents) are:<br />
<ol style="text-align: left;">
<li>Manual adjustment of the margin is <b>very unreliable</b>. Granted, other methods are prone to failing once the target web site undergoes a redesign, but here <i>even a minor change of the layout would screw the placement of your desired content and require re-adjustment</i>. What is worse, this may happen even without a website redesign proper, due to some external content (e.g. ads) affecting the elements' location and sizing. So your screen can intermittently show the wrong content, and may need frequent readjustments.</li>
<li>Even though most of the iframe's target website will be hidden, it will still be loaded and processed (all its scripts, embedded videos, plugins, and ads included), which makes it <b>very heavy</b> on the client browser (especially on older hardware such as the PlayBook). Using transformation makes matters much worse (I guess it may even force the browser to have to "invisibly" render the entire page - how else would the browser know how to scale it?). In my testing, the above example rendered my PlayBook rather unresponsive. </li>
</ol>
<div>
So, this method would be prohibitively slow for many practically relevant cases. Still, it is a simple and viable method so long as the target URL is rather lightweight and not too loaded with dynamic HTML or embedded media. A good candidate is a mobile webpage or a page specially designed to be embedded, like this: </div>
<div>
<br /></div>
<div>
<pre class="brush:html">
<div id="weather1" class="basic" style="position: relative; left: 10px; top: 10px; width: 300; height: 185px; background: white;">
<iframe scrolling="no" src="http://weather.gc.ca/wxlink/wxlink.html?cityCode=on-24&lang=e" allowtransparency="true"
style="border: 0px none; height: 185px; width: 300px;"></iframe>
</div>
</pre>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsoCJs1UOMezDgdGF2ULPVshLFhMYPtiVn4-VolevKIfK1HraAm8jv7vPEWjRLyYJjIwF-VWEGjbbdCXZYYQ4eFcDH85ZgbdozyGTShFntgtRByOXSraaTQtHltfJunKJTZnQ47V7PhKs/s1600/weathersnip2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="208" data-original-width="323" height="128" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsoCJs1UOMezDgdGF2ULPVshLFhMYPtiVn4-VolevKIfK1HraAm8jv7vPEWjRLyYJjIwF-VWEGjbbdCXZYYQ4eFcDH85ZgbdozyGTShFntgtRByOXSraaTQtHltfJunKJTZnQ47V7PhKs/s200/weathersnip2.png" width="200" /></a></div>
<br />
<br /></div>
</div>
Sergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-31337460322940365652018-01-12T19:00:00.000-05:002018-01-18T12:29:21.448-05:00Old Phone/Tablet as an Info Board Intro: Backstory and Basics<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
Some time in the past, my loving wife gave me a <a href="https://en.wikipedia.org/wiki/BlackBerry_PlayBook" target="_blank">Blackberry PlayBook</a> for my birthday. As I don't use it much these days (my smartphone has become more versatile and powerful), and would loathe to part with it (it is barely worn and beautiful, and has sentimental value), I would like to give it a second life. </div>
<div>
<br /></div>
<div>
So in a series of posts I am going to log how I turn the Playbook into an info board: an always-on, always up to date information screen next to the front door, showing some information such as the status of my commute and today's weather. </div>
<div>
<br /></div>
<div class="myaux">
Why an information screen in favor of other alternatives? For the same reasons they use departure boards at airports and transit stations: <i>it is the fastest and the least disruptive way to get the important information.</i> You can use it with your hands full (unlike your phone), and you don't have to stand there listening for a robotic voice (unlike Hey Google / Alexa / Siri).</div>
<div>
<br /></div>
<div>
Like so: (and yes, this is a sneak peek into the beta version of the end result):
<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiYubqhd3dgN4vdm0RxEDa24Ob3K4Qu54wmabJ_5qgodlBueoo9a-MESitPvRwvMALxTOoVf2UJhENwXAr7oSZSrXpB3uvtJ6_ea4hsICCdXmGOFTnKYI9PMwEm9FerJlCC5PaRSvI-Cc/s1600/READYBOARD.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiYubqhd3dgN4vdm0RxEDa24Ob3K4Qu54wmabJ_5qgodlBueoo9a-MESitPvRwvMALxTOoVf2UJhENwXAr7oSZSrXpB3uvtJ6_ea4hsICCdXmGOFTnKYI9PMwEm9FerJlCC5PaRSvI-Cc/s400/READYBOARD.png" width="400" height="295" data-original-width="549" data-original-height="405" /></a></div>
</div>
<div>
<br />
<br /></div>
<div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; position: relative; text-align: right; top: -2em;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAaAK8bQvUWx_NNYi7hq712zIoZElbuAS82YlUXgMBI3ff3ZnOrFY9uEz9OT5jwQPqG0H9yO5azk8wngZSEvvOaHIpGxw3JeXLFFFW0TMScFVTQrw_bRmgjQ8mPUq-UMQ4mqOOlXlwyrE/s1600/ds212.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img alt="" border="0" data-original-height="355" data-original-width="355" height="125" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAaAK8bQvUWx_NNYi7hq712zIoZElbuAS82YlUXgMBI3ff3ZnOrFY9uEz9OT5jwQPqG0H9yO5azk8wngZSEvvOaHIpGxw3JeXLFFFW0TMScFVTQrw_bRmgjQ8mPUq-UMQ4mqOOlXlwyrE/s320/ds212.jpg" title="" width="125" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The DiskStation server I have at home.</td></tr>
</tbody></table>
Rather than <a href="https://developer.blackberry.com/playbook/native/" target="_blank">getting the SDK</a> and writing an app (long!!!), I decided to make a web page and host it on my <a href="https://en.wikipedia.org/wiki/Synology_Inc.#Synology_DiskStation" target="_blank">DiskStation</a> (a network attached storage which has a web server function). This approach is much more versatile since it is not limited to the PlayBook or Blackberry. In fact, there will be very few PlayBook-specific points here (mostly limitations: PlayBook uses a rather old implementation of <a href="https://en.wikipedia.org/wiki/WebKit" target="_blank">Webkit</a>, and its processor is, by the modern web standards, not the fastest).<br />
<br /></div>
<div class="myaux">
On the contrary, the procedures given here should be helpful for many other devices - old iPads, old Android tables, old smartphones (even though a smartphone has a smaller screen and there will be less info that it can meaningfully display) and perhaps something even more exotic like old monitors hooked to something like a <a href="https://en.wikipedia.org/wiki/Raspberry_Pi" target="_blank">Raspberry Pi</a>.<br />
<br />
In fact, when the screen is ready, anyone at home can access it from their desktop or phone if they need the info but don't feel like physically going downstairs.
</div>
<div>
<br /></div>
<br />
So, to get the job done, we will be combining server-side programming (PHP) and client-side programming (JavaScript / jQuery).<br />
<div>
<br /></div>
Let's get started by designing an interface:<br />
<br />
<div style="font-size:small;"><pre class="brush: html">
<html>
<head>
<title>Info Screen</title>
<style>
div.background {background: black; fallback: linear-gradient(180deg, rgb(0,0,0), rgb(25,25,25)); position: absolute; left:0; top:0; height: 560px; width: 1024px; }
div.basic { background-color: white; border: 1px solid rgb(255,255,255); border-radius: 15px; overflow: hidden; padding: 5px;}
div.saver {filter:invert(100%);-webkit-filter:invert(100%);}
div.info { text-align: center; vertical-align: middle; font-family: "Arial", Helvetica, Sans-Serif; font-size:22px}
.element {font-size: 12px; text-align: center;}
.large{background-color: #ffffee;width:650px;height:400px;float:left;}
</style>
</head>
<body> <div class="background">
<div id="status" class="basic info" style="position: absolute; left: 10px; top: 10px; width: 120px; height: 90px; background: #EEFFF8; color: gray; font-size: 12px;"></div>
<div id="weather" class="basic" style="position: absolute; left: 10px; bottom: 190px; width: 120px; height: 240px; background: #EEEEFF;"></div>
<div id="hourly" class="basic info" style="position: absolute; left: 10px; bottom: 10px; width: 990px; height: 160px; background: white;"></div>
<div id="map" class="basic" style="border: none; position: absolute; right: 10px; top: 10px; width: 415px; height: 350px;"></div>
<div id="travel" class="basic info" style="position: absolute; left: 150px; bottom: 190px; width: 415px; height: 60px; background:rgb(255,255,225);"> </div>
<div id="bus" class="basic info" style="position: absolute; left: 150px; bottom: 270px; width: 415px; height: 50px; background:rgb(255,235,235);"> </div>
<div id="train1" class="basic" style="position: absolute; left: 150px; top: 10px; width: 200px; height: 195px; "></div>
<div id="train2" class="basic" style="position: absolute; left: 365px; top: 10px; width: 200px; height: 195px; "></div>
</div>
</body> </html>
</pre></div>
<br />
Note that the code above describes only the layout of the elements. In the upcoming posts (there will be 3-4 of them) I am going to discuss in detail how to actually fill the interface with contents, as well as how to script its automatic updates.<br />
<br />
<div class="myaux">
(On a side note, it looks like we've lived through a shift of paradigm about what <i>good stuff</i> is.<br />
<br />
For centuries - and indeed, persisting all the way into the end of the 20th century - the basic idea of all goods was "the good stuff is the stuff that lasts".<br />
<br />
Indeed, in a world where changes are slow and manufacturing is scarce, or expensive, or both, this made total sense. Having to replace anything (from your tools of the trade to the pair of shoes you wear) was an extra burden to be avoided if at all possible; ideally, good stuff would last a lifetime and sometimes even outlast its owners.<br />
<br />
But nowadays - at least the "modernized countries" world - this has changed. And the biggest change has been the change of pace. Stuff improves much faster than it wears out. Not only that, but also - with the digital stuff in particular - your device is only as good as the software it works with, and if that software outruns your hardware, bad luck.<br />
<br />
To summarize this in a metaphor, we became kids again - the digital kids who outgrow their digital clothes before they have any chance of wearing out. We may happen to have the best smartphone in the world today. But in a few years it will be bugged with so many "apps-that-won't update" and "sites-that-take-forever-to-load" and "services-that-no-longer-work" that using this once-perfect gadget would feel like dressing your five-year-old in clothes that were her favorite when she was three.</div>
</div>
Sergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-39577498327933165722017-11-02T11:46:00.001-04:002017-11-02T11:46:17.147-04:00Debugging without a debugger<div dir="ltr" style="text-align: left;" trbidi="on">
Another retro post, from even earlier times than the last one.<br />
<br />
<div class="myaux">
This happened during my high school years when some of my friends already owned a computer but I had none. So we used to get together after school and binge play some video games -- starting with <a href="http://spectrumcomputing.co.uk/index.php?cat=96&id=0006601" target="_blank">Lord of the Rings</a> text adventures and <a href="https://en.wikipedia.org/wiki/Laser_Squad" target="_blank">Laser Squad</a> hot-seats...<br />
<br />
...and then, rummaging through then-abundant "bootleg software shops" (a post-USSR version of Game Stop, selling bunches of floppy disks with copied games or other software and with labels hand-printed on a 9-pin dot matrix printer) I discovered <a href="https://en.wikipedia.org/wiki/Eric_the_Unready" target="_blank">Eric the Unready</a> (which you can actually play <a href="https://archive.org/details/msdos_Eric_the_Unready_1993" target="_blank">in an online emulator here</a>).<br />
<br />
I loved it at first sight. A wonderful piece of interactive fiction with hilarious jokes and puns, challenging puzzles and, myself an avid English learner at that time, an invaluable learning resource.</div>
<br />
Unfortunately, right after playing through the first chapter, we ran into something annoying: copy protection feature. One of the "Prince of Persia"-type where you would be asked a few questions and would need the original printed game manual to answer these correctly and play on.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtK7eyg_r_RkjHH0NeC0_IMlNGxe88DTweaiAu7dRC-E8zGVnsGUokzomw6CgEM8IDlWEb-xCCO7yluXCfVgDzC-vNZKdiBqp7I9Ep0vt-lVCsb5ny1TtSe_WmiqRKi0Yco7sZyxhrOIg/s1600/eric_copy.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="484" data-original-width="646" height="298" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtK7eyg_r_RkjHH0NeC0_IMlNGxe88DTweaiAu7dRC-E8zGVnsGUokzomw6CgEM8IDlWEb-xCCO7yluXCfVgDzC-vNZKdiBqp7I9Ep0vt-lVCsb5ny1TtSe_WmiqRKi0Yco7sZyxhrOIg/s400/eric_copy.png" width="400" /></a></div>
<br />
<br />
And needless to say we didn't have the manual -- and of course, neither did the shop we bought the game from.<br />
<br /> <div class="myaux">
Well, as a disclaimer... I do understand that software piracy is, in the big picture, bad bad BAD, but hey, we were 15 years old and had no idea of the big picture - nor any clue about copyrights and licensing. To us at that time, the fact that we had software on our computer meant that we could do as we pleased with it, especially given that we did buy it at a shop (sic!).<br />
<br />
And even if someone were to lecture us on the proper course of action... At that time in that part of the world, an equivalent of $30 would be a decent <i>monthly</i> salary (yes, monthly, not daily, not hourly), and there was absolutely no way an ordinary person could <i>possibly</i> make a payment anywhere to a foreign country, or for that matter, to pay in any tender other than cash - no credit cards, no wire transfers, no bank accounts...<br />
<br />
...so yes, I admit we were stealing apples from somebody else's garden, but quite unknowingly, almost unavoidably, and without causing anyone any real harm.</div><br />
<br />
<br />
But all these sentiments aside, we were already hooked, and needed a way to play on.<br />
<br />
Surely we had no Internet to look up the correct answers (there was no Google, no Chrome and even hardly any Internet Explorer yet!), we had no one to ask (the game was so out of mainstream, and the level of English command needed to play it was so untypical that we might well have been the only players in years). We also had nothing to tinker with the game with -- no <a href="https://en.wikipedia.org/wiki/Hiew" target="_blank">hiew</a>, no disassembler, no debugger proper. <br />
<br />
We did have <a href="http://www.vcfed.org/forum/showthread.php?54702-Game-wizard-32-x-dos" target="_blank">Game Wizard</a>, a utility that you would normally use to save and reload in Tetris or make yourself infinite lives in Pacman. The way you do it would be to search the memory for "3" when you have 3 lives, then for "2" when you have 2 lives, and so on; with some luck you would find the address in memory that holds the lives variable for the game. You can then set it to 99 or freeze at 3 to get infinite lives.<br />
<br />
And we gave it a try, for lack of anything better to do for the rest of the evening. We began by alternating memory searches in the state before vs. after the first question is answered -- hoping to reveal its correct answer by noticing different variable values depending on whether we chanced to answer correctly. Instead, we got:<br />
<br /><pre>
XXXX:YYYY 01 02 01 02 01 02</pre><br />
<br />
As a wild guess I just set it to 4 instead...<br />
<br />
...and got right through. (Apparently I inadvertently found the counter of the questions loop and moved right past it.) <br />All it took after that was to save the game (and the evening), granting us with endless hours of fun time.<br />
<br />
<br /></div>
Sergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.com0tag:blogger.com,1999:blog-844851375097577644.post-78187302364862551502017-11-02T11:12:00.000-04:002020-01-01T10:38:41.033-05:00The power of one-liners<div dir="ltr" style="text-align: left;" trbidi="on">
In modern software engineering where most software is written by large groups of people most of whom seem to <a href="https://blogs.msdn.microsoft.com/ericlippert/2004/06/14/reading-code-is-hard/" target="_blank">dislike reading each other's code</a>. As a result, most organizations have some <a href="https://en.wikipedia.org/wiki/Programming_style" target="_blank">style guidelines</a> for their code: proper indentation, naming conventions, commenting, you name it.<br />
<br />
And most developers will have strong opinions about how code should (and even more so, <i>shouldn't</i>) look like. And they will defend (and, their position permitting, enforce) their opinions with near-religious fervor. Most often, they will yell at you for writing this:<br />
<br />
<pre class="brush:c">for (i=0;i<n;i++) for (j=i;j<n;j++) if (a[i]>a[j]) {double t=a[i];a[i]=a[j];a[j]=t;} // bubble sort A
</pre>
<br />
<br />
and instead insist on something like this:<br />
<br />
<pre class="brush:c">// Bubble sort array A
for (counter_rows = 0; counter_rows < n; counter_rows++)
{
for (counter_columns = counter_rows; counter_columns < n; counter_columns++)
{
if (a[counter_rows] > a[counter_columns])
{
double temp_variable t = a[counter_rows];
a[counter_rows] = a[counter_columns];
a[counter_columns] = temp_variable;
}
}
}
</pre>
<br />
<br />
Yes, it looks neat and tidy. But is it really all that <i>readable</i>?<br />
<br />
No, not really.<br />
<br />
It is very hard to honestly defend a standpoint that a piece of code is easier to read if have to scroll through three screens to read it, as opposed to fitting it on one screen. "But this way it is more organized", comes the objection. Very true, but <i>how</i> is it organized?<br />
<br />
It is organized <b>by instructions</b>.<br />
<br />
But what for? By instruction is how the <i>compiler</i> looks at the code, but the compiler does not give the slightest damn about your code style. Humans are much more interested in <i>what</i> the code does than in <i>how</i> the code does it (assuming that you, a fellow developer, already have some knowledge of the "how" once you know the "what").<br />
<br />
From this standpoint it makes much more sense to organize the code <i>by logically distinct blocks </i>-- important steps of your algorithm that you would put on your flowchart or pseudocode (pseudocode, after all, was invented specifically for this purpose: convey the meaning of complicated code in a simpler, readable, understandable form).<br />
<br />
And this means that simple, elegant one-liners -- <i>once (and if) they are self-evident in what they do</i> -- are much more preferable than expanding them on two screens. See for yourselves:<br />
<br />
<div style="font-size: 75%;">
<pre class="brush:c">//simple operations, e.g. sumproduct or matrix multiplication
double S=0; for (int i=0;i<N;i++) S+=a[i]*b[i];
for (int i=0;i<N;i++) for (int j=0;j<N;j++) for (int k=0;k<N;k++) c[i][j]+=a[i][k]*b[k][j];
// one-liner error checking, prep or boilerplate
if (failed || !solution_good) return false;
double param; if (!genericParam.Has_Double) throw Error; param = genericParam.Get_Double();
customVector<double> vec(vec1); int vec_n=vec.Count(); double* vec_ptr=vec.Data();
//getter/setter methods
double someClass::getSomeProperty() {return someProperty;}
void someClass::setSomeProperty(double arg) {someProperty=arg;}
//etc...
</pre>
</div>
<br />
<br />
<div class="myaux">
<b>UPDATE</b>: Following some discussion, I feel I need to clear up a confusion here. Any code, prettified, is more readable than <i>the same code</i>, minified. The idea of one-liners isn't about improving the readability of the one-liner code <b>itself</b>! It is about the exact opposite: the one-liner code is assumed to be <i>trivial</i> and therefore <i>not worth going into any great detail about</i>, so the idea is to minify the one-liner code so that it does not get in the way of what's really important and interesting in your code. In other words, it is about improving the readability of the code <b>surrounding</b> your one-liners.
</div>
<br />
<br />
As for naming conventions, they definitely make sense for anything that would be (re)used in several places through the code. If something is set up in one place and is used elsewhere, by all means make the name of the variable (class, object, ...) speak for itself.<br />
<br />
That said, <i>still</i> try to keep it short. Calculations in the code are formulas, and formulas read much easier with shorter variables than with long, verbose ones; <span class="myinline"><i>that's why they introduced variables</i> in textbook formulas in the first place, and they do write <i>E = mgh</i> instead of "Potential_energy = mass * specific_gravity * distance" anywhere beyond grade two at school.</span><br />
<br />
For intermediate variables such as loop counters, simply don't bother. Mathematical names such as <i>a, x, y, i, j, k </i>will perfectly do and they will make your calculations so much easier.<br />
<br />
<div class="myaux">
There are exceptions -- sometimes, when naming is especially prone to confusion, do add some mnemonics, such as <i>i_row</i> and <i>i_col</i> rather than <i>i</i> and <i>j</i>, lest you mess up your array indices. But in most cases, formulas in the code need not look any more verbose than they do on your scrap paper. Sometimes, it is even advisable to assign "long" mnemonic variables to short ones, do the math, and then assign the result back to the long variables.</div>
<br />
<br />
So -- no, I'm not saying your code should look like <a href="http://www.nanochess.org/chess3.html" target="_blank">Toledo Picochess</a> (see below). But <b>do write in your IDE as you would write on the blackboard, and do </b><b>write code as you would write pseudocode.</b><br />
<br />
After all, making code readable is all about making it readable for a <i>human</i>.<br />
<br />
<br />
<div class="myaux" style="font-size: 80%;">
P.S. Toledo Picochess looks like this: (now THAT's truly unreadable code!)
<br />
<pre class="brush: c">#define F (getchar()&15)
#define v main(0,0,0,0,
#define Z while(
#define P return y=~y,
#define _ ;if(
char*l="dbcefcbddabcddcba~WAB+ +BAW~ +-48HLSU?A6J57IKJT576,";B,y,
b,I[149];main(w,c,h,e,S,s){int t,o,L,E,d,O=*l,N=-1e9,p,*m=I,q,r,x=10 _*I){y=~y;
Z--O>20){o=I[p=O]_ q=o^y,q>0){q+=(q<2)*y,t=q["51#/+++"],E=q["95+3/33"];do{r=I[p
+=t[l]-64]_!w|p==w&&q>1|t+2<E|!r){d=abs(O-p)_!r&(q>1|d%x<1)|(r^y)<-1){_(r^y)<-6
)P 1e5-443*h;O[I]=0,p[I]=q<2&(89<p|30>p)?5^y:o;L=(q>1?6-q?l[p/x-1]-l[O/x-1]-q+2
:0:(p[I]-o?846:d/8))+l[r+15]*9-288+l[p%x]-h-l[O%x];L-=s>h||s==h&L>49&1<s?main(s
>h?0:p,L,h+1,e,N,s):0 _!(B-O|h|p-b|S|L<-1e4))return 0;O[I]=o,p[I]=r _ S|h&&(L>N
||!h&L==N&&1&rand())){N=L _!h&&s)B=O,b=p _ h&&c-L<S)P N;}}}t+=q<2&t+3>E&((y?O<
80:39<O)||r);}Z!r&q>2&q<6||(p=O,++t<E));}}P N+1e9?N:0;}Z I[B]=-(21>B|98<B|2>(B+
1)%x),++B<120);Z++m<9+I)30[m]=1,90[m]=~(20[m]=*l++&7),80[m]=-2;Z p=19){Z++p<O)
putchar(p%x-9?"KQRBNP .pnbrqk"[7+p[I]]:x)_ x-(B=F)){B+=O-F*x;b=F;b+=O-F*x;Z x-F
);}else v 1,3+w);v 0,1);}}
</pre>
</div>
<br /></div>
Sergei http://www.blogger.com/profile/03424199158724870067noreply@blogger.com0