Jekyll2023-08-02T12:19:32+00:00https://roland-ewald.github.io/feed.xmlroland-ewald.github.ioCompiling plv8 extension for PostgreSQL 122020-11-13T00:00:00+00:002020-11-13T00:00:00+00:00https://roland-ewald.github.io/2020/11/13/postgresql-plv8<p>There are no Linux binaries available for recent versions of the <a href="https://github.com/plv8/plv8">plv8</a> extesion, which adds JavaScript support to PostreSQL.
Compiling the extension manually is a little tricky, because additional dependencies are necessary (discussed <a href="https://github.com/plv8/plv8/issues/283#issuecomment-397359439">here</a>), while no copy of the V8 library, such as <code class="language-plaintext highlighter-rouge">libnode-dev</code>, must be installed (discussed <a href="https://github.com/plv8/plv8/issues/387#issuecomment-605663975">here</a>). The build also relies on <a href="https://xkcd.com/353/">python 2</a>. This is how it should work on a Ubuntu 20.10 setup:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># See https://github.com/plv8/plv8/issues/387#issuecomment-605663975 for all libraries that may be an issue:</span>
<span class="nb">sudo </span>apt-get remove libnode libnode-dev
<span class="c"># See https://github.com/plv8/plv8/issues/283#issuecomment-397359439</span>
<span class="nb">sudo </span>apt-get <span class="nb">install </span>python postgresql-server-dev-12 make git pkg-config chromium-browser subversion clang apg ninja-build cmake libc++-dev libc++abi-dev
wget https://github.com/plv8/plv8/archive/v2.3.15.tar.gz
<span class="nb">tar </span>xfvz v2.3.15.tar.gz
<span class="nb">cd </span>plv8...
<span class="nb">sudo </span>make
<span class="nb">sudo </span>make <span class="nb">install
</span>make installcheck
</code></pre></div></div>
<p>Enjoy!</p>There are no Linux binaries available for recent versions of the plv8 extesion, which adds JavaScript support to PostreSQL. Compiling the extension manually is a little tricky, because additional dependencies are necessary (discussed here), while no copy of the V8 library, such as libnode-dev, must be installed (discussed here). The build also relies on python 2. This is how it should work on a Ubuntu 20.10 setup:IntelliJ IDEA plugin for bulk-renaming Java types2020-06-01T00:00:00+00:002020-06-01T00:00:00+00:00https://roland-ewald.github.io/2020/06/01/intellij-bulk-rename<p>Last week I wanted to rename a larger amount of Java types so that they are consistent with a new naming scheme.
This does not seem to be easily possible in IntelliJ, so I wrote a <a href="https://github.com/roland-ewald/intellij-bulk-rename/">small plugin</a> that imports a CSV file with
the ‘refactoring instructions’ and runs them in bulk.
It also allows to define type-specific refactoring options, such as telling IntelliJ to also replace occurrences of the type name in other text files.</p>
<p>The developers at JetBrains really put some effort into supporting plugin developers. There are many helpful context-specific messages,
e.g. to inform you about APIs that are soon to be deprecated, or what exactly you forgot to put into your <code class="language-plaintext highlighter-rouge">plugin.xml</code>.
The latter even comes with a custom auto-complete for module dependencies.
There were a few minor roadblocks<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>, but nothing serious. This is the power of dogfooding, I guess.</p>
<h3 id="downloader-beware">Downloader beware</h3>
<p>This was a <em>weekend project</em>, so while I’m happy with the refactoring result itself, there are many rough edges (and I’m pretty sure there are better ways to use the IntelliJ API).
Nevertheless, it may be a useful blueprint for implementing similar utilities that help with larger ‘one-off’ refactorings.</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>For example, the predefined gradle setup requires more than the default RAM on my machine, so a <a href="https://github.com/roland-ewald/intellij-bulk-rename/blob/fafea9d8c7dc9cfc190db517fdcb9e04ed03d506/gradle.properties#L4">custom setting in the <code class="language-plaintext highlighter-rouge">gradle.properties</code> file</a> was required to initially configure the project. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Last week I wanted to rename a larger amount of Java types so that they are consistent with a new naming scheme. This does not seem to be easily possible in IntelliJ, so I wrote a small plugin that imports a CSV file with the ‘refactoring instructions’ and runs them in bulk. It also allows to define type-specific refactoring options, such as telling IntelliJ to also replace occurrences of the type name in other text files.Spring Boot projects in IntelliJ IDEA2018-04-01T00:00:00+00:002018-04-01T00:00:00+00:00https://roland-ewald.github.io/2018/04/01/idea-gradle-spring-boot<h2 id="problem">Problem</h2>
<p>Setting up <a href="https://gradle.org/">gradle-based</a> <a href="https://github.com/spring-projects/spring-boot">Spring Boot</a> projects in <a href="https://www.jetbrains.com/idea/">IntelliJ IDEA</a> is tricky.</p>
<p>A good development experience means it is easy to</p>
<ul>
<li>
<p>start any <code class="language-plaintext highlighter-rouge">@SpringBootApplication</code> (via its <code class="language-plaintext highlighter-rouge">main</code> method)</p>
</li>
<li>
<p>start any test case, including integration tests using <code class="language-plaintext highlighter-rouge">@SpringBootTest</code> etc.</p>
</li>
</ul>
<p>Since local development machines cannot reach all remote services and often rely on mock-up beans for those (which, of course, are part of the <code class="language-plaintext highlighter-rouge">test</code> code), and also rely on <code class="language-plaintext highlighter-rouge">.properties</code> files for application profiles such as <code class="language-plaintext highlighter-rouge">test</code> or <code class="language-plaintext highlighter-rouge">dev</code> (which, of course, are part of the <code class="language-plaintext highlighter-rouge">test</code> resources), you may run into trouble when trying to run applications and tests in IDEA.</p>
<p>This is because IDEA <a href="https://youtrack.jetbrains.com/issue/IDEA-160167">does not allow</a> simple ad-hoc modifications to the classpath, and at the same time will override all manually changed dependencies whenever you touch a <code class="language-plaintext highlighter-rouge">build.gradle</code> file <em>and</em> will also not honor the convention of adding test code and test resources to the Java classpath<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>.</p>
<h2 id="solutions">Solutions</h2>
<p>To solve this, there are several options:</p>
<ul>
<li>
<p>You can manually configure your IDEA module to also depend on the test sources and all resource directories. This is only feasible when the gradle setup rarely changes, because the dependencies will be reset when the gradle project is updated.</p>
</li>
<li>
<p>You can <a href="http://mrhaki.blogspot.de/2016/11/gradle-goodness-delegate-build-and-run.html">delegate IDEA build and run tasks to Gradle</a>, which works but is rather slow.</p>
</li>
<li>
<p>You can play around with custom class loaders or <a href="https://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html"><code class="language-plaintext highlighter-rouge">-Xbootclasspath/a:</code></a>, and the like. However, this is also tricky and makes the whole setup more fragile.</p>
</li>
<li>
<p>You can play around with the raw XML that define the IDEA modules and add dependencies there (or use the <code class="language-plaintext highlighter-rouge">idea</code> gradle plugin’s <a href="https://docs.gradle.org/current/userguide/idea_plugin.html"><code class="language-plaintext highlighter-rouge">ẁhenMerged</code></a> for this). Again, this adds complexity and makes the software fragile (because newer IDEA versions may also have another XML format).</p>
</li>
</ul>
<h2 id="simpler-solution">Simpler Solution</h2>
<p>There is no perfect solution for this, but if you do <em>not</em> rely on <a href="https://dzone.com/articles/resource-filtering-gradle">gradle’s resource processing features</a><sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup> you can just throwing ‘everything’ together, e.g. via</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">allprojects</span> <span class="o">{</span>
<span class="n">apply</span> <span class="nl">plugin:</span> <span class="s1">'idea'</span>
<span class="n">idea</span> <span class="o">{</span>
<span class="n">module</span> <span class="o">{</span>
<span class="n">inheritOutputDirs</span> <span class="o">=</span> <span class="kc">false</span>
<span class="n">outputDir</span> <span class="nf">file</span><span class="o">(</span><span class="s2">"$buildDir/intellij"</span><span class="o">)</span>
<span class="n">testOutputDir</span> <span class="nf">file</span><span class="o">(</span><span class="s2">"$buildDir/intellij"</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This is easy to set up (should be picked up by IDEA automatically, otherwise just run <code class="language-plaintext highlighter-rouge">gradle idea</code>), easy to debug (if a resource or bean in the same module cannot be found, just check the contents of<code class="language-plaintext highlighter-rouge">build/intellij</code>), and easy to use (it is separate from your gradle build output).</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Unfortunately, this is not configurable either. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>Actually a nice feature, and still useful in all other kinds of files that are not crucial for your Spring Boot setup. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>ProblemWant to comment? Write an email.2017-09-26T00:00:00+00:002017-09-26T00:00:00+00:00https://roland-ewald.github.io/2017/09/26/no-comment<p>I could confirm the results <a href="https://notes.ayushsharma.in/2017/09/im-killing-disqus-comments-on-my-blog-heres-why">posted by Ayush Sharma</a> (and featured on <a href="https://news.ycombinator.com/item?id=15334207">HN</a>) regarding the large performance hit a website takes when loading <a href="https://disqus.com/">Disqus</a> comments, apparently because they collaborate closely with some user-tracking ads company.</p>
<p>This page, for example, loads <em>10 times slower</em> when tested with <a href="https://gtmetrix.com">GTmetrix</a> (not sure how objective this is): <code class="language-plaintext highlighter-rouge">0.7</code> seconds vs. <code class="language-plaintext highlighter-rouge">7.2</code> seconds.</p>
<p>This is quite a lot, and the over 70 redirects they generate, e.g. to the following domains, are certainly not helping:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>https://aa.agkn.com
https://beacon.krxd.net
https://cm.g.doubleclick.net
https://d.agkn.com
https://dpm.demdex.net
https://e.nexac.com
https://ei.rlcdn.com
https://i.liadm.com
https://ib.adnxs.com
https://idsync.rlcdn.com
https://io.narrative.io
https://licensebuttons.net
https://loadus.exelator.com
https://p.adsymptotic.com
https://pippio.com
https://pixel.tapad.com
https://pm.w55c.net
https://rc.rlcdn.com
https://secure.insightexpressai.com
https://sp.adbrn.com
https://stags.bluekai.com
https://staticxx.facebook.com
https://sync.mathtag.com
https://tags.bluekai.com
https://usermatch.krxd.net
https://www.facebook.com
https://x.bidswitch.net
https://x.dlx.addthis.com
</code></pre></div></div>
<p>This is on top of all the privacy and security implications, and since nobody comments here anyway I will just disable them.</p>
<p>If you absolutely need to tell me I’m wrong, you can just write an e-mail :-)</p>I could confirm the results posted by Ayush Sharma (and featured on HN) regarding the large performance hit a website takes when loading Disqus comments, apparently because they collaborate closely with some user-tracking ads company.Query AWS CloudWatch in Java2017-07-09T00:00:00+00:002017-07-09T00:00:00+00:00https://roland-ewald.github.io/2017/07/09/aws-cloudwatch-auto-throttling<p>If you run a service on the <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html">T2 tier of AWS</a>, your machines have a specific compute budget. So, for example, you may want to delay certain CPU-intensive background tasks to a later time, throttling them by the <code class="language-plaintext highlighter-rouge">CPUCreditBalance</code> your machine currently has.</p>
<p>The current (averaged) <code class="language-plaintext highlighter-rouge">CPUCreditBalance</code> can be retrieved via <a href="https://aws.amazon.com/cloudwatch/">CloudWatch</a>, but I found the Java API rather unforgiving<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> – it fails silently when certain elements like the unit (<code class="language-plaintext highlighter-rouge">StandardUnit.Count</code>) are missing from the API call. Here is a snippet that retrieves the last (average) <code class="language-plaintext highlighter-rouge">CPUCreditBalance</code>, as a starting point:</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nc">String</span> <span class="n">ec2InstanceId</span> <span class="o">=</span> <span class="nc">EC2MetadataUtils</span><span class="o">.</span><span class="na">getInstanceId</span><span class="o">();</span>
<span class="nc">LocalDateTime</span> <span class="n">endTime</span> <span class="o">=</span> <span class="nc">LocalDateTime</span><span class="o">.</span><span class="na">now</span><span class="o">();</span>
<span class="nc">LocalDateTime</span> <span class="n">startTime</span> <span class="o">=</span> <span class="n">endTime</span><span class="o">.</span><span class="na">minusMinutes</span><span class="o">(</span><span class="mi">120</span><span class="o">);</span>
<span class="nc">GetMetricStatisticsRequest</span> <span class="n">cpuCreditBalanceRequest</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">GetMetricStatisticsRequest</span><span class="o">()</span>
<span class="o">.</span><span class="na">withNamespace</span><span class="o">(</span><span class="s">"AWS/EC2"</span><span class="o">)</span>
<span class="o">.</span><span class="na">withStatistics</span><span class="o">(</span><span class="nc">Statistic</span><span class="o">.</span><span class="na">Average</span><span class="o">)</span>
<span class="o">.</span><span class="na">withMetricName</span><span class="o">(</span><span class="s">"CPUCreditBalance"</span><span class="o">)</span>
<span class="o">.</span><span class="na">withPeriod</span><span class="o">(</span><span class="mi">60</span><span class="o">)</span>
<span class="o">.</span><span class="na">withUnit</span><span class="o">(</span><span class="nc">StandardUnit</span><span class="o">.</span><span class="na">Count</span><span class="o">)</span>
<span class="o">.</span><span class="na">withStartTime</span><span class="o">(</span><span class="nc">Date</span><span class="o">.</span><span class="na">from</span><span class="o">(</span><span class="n">startTime</span><span class="o">.</span><span class="na">toInstant</span><span class="o">(</span><span class="nc">ZoneOffset</span><span class="o">.</span><span class="na">UTC</span><span class="o">)))</span>
<span class="o">.</span><span class="na">withEndTime</span><span class="o">(</span><span class="nc">Date</span><span class="o">.</span><span class="na">from</span><span class="o">(</span><span class="n">endTime</span><span class="o">.</span><span class="na">toInstant</span><span class="o">(</span><span class="nc">ZoneOffset</span><span class="o">.</span><span class="na">UTC</span><span class="o">)))</span>
<span class="o">.</span><span class="na">withDimensions</span><span class="o">(</span><span class="k">new</span> <span class="nc">Dimension</span><span class="o">().</span><span class="na">withName</span><span class="o">(</span><span class="s">"InstanceId"</span><span class="o">).</span><span class="na">withValue</span><span class="o">(</span><span class="n">ec2InstanceId</span><span class="o">));</span>
<span class="nc">GetMetricStatisticsResult</span> <span class="n">cpuCreditBalanceResult</span> <span class="o">=</span> <span class="n">getAmazonCloudWatch</span><span class="o">().</span><span class="na">get</span><span class="o">().</span><span class="na">getMetricStatistics</span><span class="o">(</span><span class="n">cpuCreditBalanceRequest</span><span class="o">);</span>
<span class="nc">List</span><span class="o"><</span><span class="nc">Datapoint</span><span class="o">></span> <span class="n">metricData</span> <span class="o">=</span> <span class="n">cpuCreditBalanceResult</span><span class="o">.</span><span class="na">getDatapoints</span><span class="o">();</span>
<span class="k">if</span> <span class="o">(</span><span class="n">metricData</span> <span class="o">==</span> <span class="kc">null</span> <span class="o">||</span> <span class="n">metricData</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">())</span> <span class="o">{</span>
<span class="no">LOG</span><span class="o">.</span><span class="na">warn</span><span class="o">(</span><span class="s">"Metric 'CPUCreditBalance' is not available."</span><span class="o">);</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="nc">Datapoint</span> <span class="n">mostRecentData</span> <span class="o">=</span> <span class="n">metricData</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">metricData</span><span class="o">.</span><span class="na">size</span><span class="o">()</span> <span class="o">-</span> <span class="mi">1</span><span class="o">);</span>
<span class="nc">Double</span> <span class="n">result</span> <span class="o">=</span> <span class="n">mostRecentData</span><span class="o">.</span><span class="na">getAverage</span><span class="o">();</span>
<span class="no">LOG</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">"Latest average of 'CPUCreditBalance': "</span> <span class="o">+</span> <span class="n">result</span><span class="o">);</span>
<span class="o">}</span></code></pre></figure>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>In any case, before using the Java API, I would always suggest to try out the request via the <a href="https://aws.amazon.com/cli/">AWS CLI</a> first; this is a complex API with many features. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>If you run a service on the T2 tier of AWS, your machines have a specific compute budget. So, for example, you may want to delay certain CPU-intensive background tasks to a later time, throttling them by the CPUCreditBalance your machine currently has.A (very) simple R -> Python integration2017-03-04T00:00:00+00:002017-03-04T00:00:00+00:00https://roland-ewald.github.io/2017/03/04/simple-r-python-integration<p>For a side project, I needed to run a piece of Python code from R. Given their popularity (e.g., they hold the spots #2 and #9 in <a href="https://pypl.github.io/PYPL.html">the PYPL popularity of programming languages index</a>) and their focus on being pragmatic, I thought this should be very simple.</p>
<p>I was wrong. There seems to be an easy solution to call R from Python (via <a href="https://rpy2.bitbucket.io/">rpy</a>) but the other way around turns out to be much more difficult. The packages that promise to do so (such as <a href="https://cran.r-project.org/web/packages/rPython/index.html">rPython</a> or <a href="https://github.com/cjgb/rPython-win">rPythonWin</a>) seem to be not really in widespread use, and are thus not maintained very well<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>.</p>
<p>I shouldn’t complain about open source (and send patches instead ;-), but there seems to be no well-maintained, easy-to-use, cross-platform R plugin that allows to call Python. Since I only have to make a few calls to a single Python function and runtime performance is not an issue, I gave up and simply used JSON and the command line for data transfer and invocation.</p>
<p>Here are some code samples that should get you going, if you also just want to cobble something together without being particularly proficient in either language.</p>
<p>The R snippet (requires an <code class="language-plaintext highlighter-rouge">install.packages("rjson")</code> in the R console):</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">library</span><span class="p">(</span><span class="n">rjson</span><span class="p">)</span><span class="w">
</span><span class="n">python_caller</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="k">function</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="p">,</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">INPUT_FILE</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="s1">'python_input.json'</span><span class="w">
</span><span class="n">OUTPUT_FILE</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="s1">'python_output.json'</span><span class="w">
</span><span class="n">PYTHON_FILE</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="s1">'my_code.py'</span><span class="w">
</span><span class="c1"># Write input parameters to JSON file</span><span class="w">
</span><span class="n">json</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">toJSON</span><span class="p">(</span><span class="nf">list</span><span class="p">(</span><span class="w">
</span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">x</span><span class="p">,</span><span class="w">
</span><span class="n">y</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">y</span><span class="p">,</span><span class="w">
</span><span class="n">z</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">z</span><span class="p">))</span><span class="w">
</span><span class="n">fileConn</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">file</span><span class="p">(</span><span class="n">INPUT_FILE</span><span class="p">)</span><span class="w">
</span><span class="n">writeLines</span><span class="p">(</span><span class="n">json</span><span class="p">,</span><span class="w"> </span><span class="n">fileConn</span><span class="p">)</span><span class="w">
</span><span class="n">close</span><span class="p">(</span><span class="n">fileConn</span><span class="p">)</span><span class="w">
</span><span class="c1"># Run Python code</span><span class="w">
</span><span class="n">system</span><span class="p">(</span><span class="n">paste</span><span class="p">(</span><span class="s1">'python '</span><span class="p">,</span><span class="w"> </span><span class="n">PYTHON_FILE</span><span class="p">))</span><span class="w">
</span><span class="c1">#Read results from JSON file</span><span class="w">
</span><span class="n">json</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">readChar</span><span class="p">(</span><span class="n">OUTPUT_FILE</span><span class="p">,</span><span class="w"> </span><span class="n">file.info</span><span class="p">(</span><span class="n">OUTPUT_FILE</span><span class="p">)</span><span class="o">$</span><span class="n">size</span><span class="p">)</span><span class="w">
</span><span class="n">result</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">fromJSON</span><span class="p">(</span><span class="n">json</span><span class="p">)</span><span class="w">
</span><span class="n">file.remove</span><span class="p">(</span><span class="n">OUTPUT_FILE</span><span class="p">)</span><span class="w">
</span><span class="c1"># Return values</span><span class="w">
</span><span class="nf">return</span><span class="p">(</span><span class="nf">list</span><span class="p">(</span><span class="n">result1</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">result</span><span class="o">$</span><span class="s1">'result1'</span><span class="p">,</span><span class="w"> </span><span class="n">result2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">result</span><span class="o">$</span><span class="s1">'result2'</span><span class="p">))</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<p>The Python snippet for <code class="language-plaintext highlighter-rouge">my_code.py</code>:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="c1"># Read input parameters from JSON file
</span><span class="n">INPUT_FILE</span> <span class="o">=</span> <span class="s">'python_input.json'</span>
<span class="n">OUTPUT_FILE</span> <span class="o">=</span> <span class="s">'python_output.json'</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">INPUT_FILE</span><span class="p">)</span> <span class="k">as</span> <span class="n">json_file</span><span class="p">:</span>
<span class="n">my_args</span> <span class="o">=</span> <span class="n">json</span><span class="p">.</span><span class="n">load</span><span class="p">(</span><span class="n">json_file</span><span class="p">)</span>
<span class="n">os</span><span class="p">.</span><span class="n">remove</span><span class="p">(</span><span class="n">INPUT_FILE</span><span class="p">)</span>
<span class="c1"># Run some actual code here instead:
</span><span class="n">output</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">output</span><span class="p">[</span><span class="s">"result1"</span><span class="p">]</span> <span class="o">=</span> <span class="n">my_args</span><span class="p">[</span><span class="s">'x'</span><span class="p">]</span> <span class="o">+</span> <span class="n">my_args</span><span class="p">[</span><span class="s">'y'</span><span class="p">]</span> <span class="o">-</span> <span class="n">my_args</span><span class="p">[</span><span class="s">'z'</span><span class="p">]</span>
<span class="n">output</span><span class="p">[</span><span class="s">"result2"</span><span class="p">]</span> <span class="o">=</span> <span class="n">my_args</span><span class="p">[</span><span class="s">'x'</span><span class="p">]</span> <span class="o">-</span> <span class="n">my_args</span><span class="p">[</span><span class="s">'y'</span><span class="p">]</span> <span class="o">+</span> <span class="n">my_args</span><span class="p">[</span><span class="s">'z'</span><span class="p">]</span>
<span class="c1"># Write results to JSON file
# Note that some Python objects (e.g. NumPy arrays) need to be converted before writing them (e.g. see http://stackoverflow.com/a/32850511/109942)
</span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">OUTPUT_FILE</span><span class="p">,</span> <span class="s">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">json</span><span class="p">.</span><span class="n">dump</span><span class="p">(</span><span class="n">output</span><span class="p">,</span> <span class="n">f</span><span class="p">)</span></code></pre></figure>
<p>When calling</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">python_caller</span><span class="p">(</span><span class="m">1</span><span class="p">,</span><span class="m">2</span><span class="p">,</span><span class="m">3</span><span class="p">)</span></code></pre></figure>
<p>in the R console, the intermediate files <code class="language-plaintext highlighter-rouge">python_input.json</code> and <code class="language-plaintext highlighter-rouge">python_output.json</code> look as expected (before being deleted):</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">python_input.json</code>:</li>
</ul>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="nl">"x"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nl">"y"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nl">"z"</span><span class="p">:</span><span class="mi">3</span><span class="p">}</span></code></pre></figure>
<ul>
<li><code class="language-plaintext highlighter-rouge">python_output.json</code>:</li>
</ul>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="nl">"result2"</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="nl">"result1"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">}</span></code></pre></figure>
<p>This approach is far from perfect (for example: no debug mode that keeps the intermediate files, no support for multi-threading, assumes <code class="language-plaintext highlighter-rouge">python</code> to be available on the <code class="language-plaintext highlighter-rouge">PATH</code>, python command-line options like <code class="language-plaintext highlighter-rouge">-O</code> are missing), but it is also simple enough to understand this again in a few months, it requires no extra setup, and it works cross-platform (I hope ;-).</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>This is a euphemism for me not being able to set them up, neither on Windows nor on Linux, and neither with a current R version (3.3.2) nor an old one (2.15.1) that they should support (according to their documentation). <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>For a side project, I needed to run a piece of Python code from R. Given their popularity (e.g., they hold the spots #2 and #9 in the PYPL popularity of programming languages index) and their focus on being pragmatic, I thought this should be very simple.Configuring terminal usage in Atom2017-02-26T00:00:00+00:002017-02-26T00:00:00+00:00https://roland-ewald.github.io/2017/02/26/atom-terminal<p>I re-visited the <a href="https://atom.io">Atom</a> editor in the last weeks; after all <a href="https://github.com/atom/atom/releases/tag/v1.14.0">its performance is being improved step by step</a><sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> and by now the development speed seems to have settled down a little.</p>
<p>Overall, I am quite impressed with how simple the editor is to configure<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup>.</p>
<p>For example, I ran into a problem when configuring terminal support. While the <a href="https://atom.io/packages/terminal-fusion">terminal-fusion</a> package (a Linux-only fork of <a href="https://github.com/platformio/platformio-atom-ide-terminal">platformio-atom-ide-terminal</a>) works well out-of-the-box, it seems to miss a simple way to switch between terminal window and the current editor pane (and to auto-hide the terminal if it is not in focus).</p>
<p>Luckily, there is a <a href="https://github.com/platformio/platformio-atom-ide-terminal/issues/62#issuecomment-252450925">code snippet for <code class="language-plaintext highlighter-rouge">platformio-atom-ide-terminal</code></a><sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup> that just needs a little tinkering to make it compatible with <code class="language-plaintext highlighter-rouge">terminal-fusion</code>, and then goes into <code class="language-plaintext highlighter-rouge">init.coffee</code>:</p>
<figure class="highlight"><pre><code class="language-coffee" data-lang="coffee"><span class="nx">atom</span><span class="p">.</span><span class="na">packages</span><span class="p">.</span><span class="na">onDidActivatePackage</span> <span class="p">(</span><span class="nx">pack</span><span class="p">)</span> <span class="o">-></span>
<span class="k">if</span> <span class="nx">pack</span><span class="p">.</span><span class="na">name</span> <span class="o">==</span> <span class="s">'terminal-fusion'</span>
<span class="nx">atom</span><span class="p">.</span><span class="na">commands</span><span class="p">.</span><span class="na">add</span> <span class="s">'atom-workspace'</span><span class="p">,</span>
<span class="s">'editor:focus-main'</span><span class="p">,</span> <span class="o">-></span>
<span class="nx">p</span> <span class="o">=</span> <span class="nx">atom</span><span class="p">.</span><span class="na">workspace</span><span class="p">.</span><span class="na">getActivePane</span><span class="p">()</span>
<span class="nx">panels</span> <span class="o">=</span> <span class="nx">atom</span><span class="p">.</span><span class="na">workspace</span><span class="p">.</span><span class="na">getBottomPanels</span><span class="p">()</span>
<span class="nx">term</span> <span class="o">=</span> <span class="nx">panels</span><span class="p">.</span><span class="na">find</span> <span class="p">(</span><span class="nx">pan</span><span class="p">)</span> <span class="o">-></span>
<span class="nx">pan</span><span class="p">.</span><span class="na">item</span><span class="p">.</span><span class="na">constructor</span><span class="p">.</span><span class="na">name</span> <span class="o">==</span> <span class="s">'TerminalFusionView'</span>
<span class="k">if</span> <span class="o">not</span> <span class="nx">term</span>
<span class="c1"># Open a new terminal</span>
<span class="nx">editor</span> <span class="o">=</span> <span class="nx">atom</span><span class="p">.</span><span class="na">workspace</span><span class="p">.</span><span class="na">getActiveTextEditor</span><span class="p">()</span>
<span class="nx">atom</span><span class="p">.</span><span class="na">commands</span><span class="p">.</span><span class="na">dispatch</span><span class="p">(</span><span class="nx">atom</span><span class="p">.</span><span class="na">views</span><span class="p">.</span><span class="na">getView</span><span class="p">(</span><span class="nx">editor</span><span class="p">),</span> <span class="s">'terminal-fusion:new'</span><span class="p">)</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">term</span> <span class="o">and</span> <span class="nx">p</span><span class="p">.</span><span class="na">focused</span>
<span class="nx">term</span><span class="p">.</span><span class="na">item</span><span class="p">.</span><span class="na">open</span><span class="p">()</span>
<span class="nx">term</span><span class="p">.</span><span class="na">item</span><span class="p">.</span><span class="na">focus</span><span class="p">()</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">term</span> <span class="o">and</span> <span class="o">!</span><span class="nx">p</span><span class="p">.</span><span class="na">focused</span>
<span class="nx">term</span><span class="p">.</span><span class="na">hide</span><span class="p">()</span>
<span class="nx">p</span><span class="p">.</span><span class="na">activate</span><span class="p">()</span></code></pre></figure>
<p>It’s easy to understand what is going on here, even without knowing much about <a href="http://coffeescript.org/">CoffeeScript</a>.</p>
<p>Now just add this shortcut definition to your <code class="language-plaintext highlighter-rouge">keymap.cson</code>:</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s1">'</span><span class="s">.platform-linux,</span><span class="nv"> </span><span class="s">atom-text-editor</span><span class="nv"> </span><span class="s">atom-workspace'</span><span class="pi">:</span>
<span class="s1">'</span><span class="s">ctrl-ä'</span><span class="pi">:</span> <span class="s1">'</span><span class="s">editor:focus-main'</span></code></pre></figure>
<p>And yes, this is an <code class="language-plaintext highlighter-rouge">ä</code> you see there – change as appropriate if you are not blessed with a German keyboard layout :-)</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>My impression so far: absolutely usable for ‘normal’ documents (e.g. code), but still <em>much</em> slower than <a href="https://www.sublimetext.com">Sublime Text</a> on larger files (e.g. log files of a few megabytes, or files with very long lines – although that seems to get fixed in <a href="https://github.com/atom/atom/releases/tag/v1.15.0-beta3">v1.15</a>). Also, starting the editor takes a few seconds, so it’s a little too slow for quick one-off edit tasks (git commits etc.). <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>Of course it helps that important shortcuts like <code class="language-plaintext highlighter-rouge">ctrl-shift-p</code> are consistent with Sublime Text :-) <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>The last version of that snippet, on which the above code is based, <a href="https://github.com/platformio/platformio-atom-ide-terminal/issues/62#issuecomment-263085500">can be found here</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>I re-visited the Atom editor in the last weeks; after all its performance is being improved step by step1 and by now the development speed seems to have settled down a little. My impression so far: absolutely usable for ‘normal’ documents (e.g. code), but still much slower than Sublime Text on larger files (e.g. log files of a few megabytes, or files with very long lines – although that seems to get fixed in v1.15). Also, starting the editor takes a few seconds, so it’s a little too slow for quick one-off edit tasks (git commits etc.). ↩PostgreSQL integration testing troubles2017-02-19T00:00:00+00:002017-02-19T00:00:00+00:00https://roland-ewald.github.io/2017/02/19/postrges-jpa-jdbc<p>I had quite some trouble this weekend to get a database integration test suite running with current versions of Spring Boot (1.5.1) and PostgreSQL (9.6.1).</p>
<p>There was this one error that would pop up sporadically:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">org.postgresql.util.PSQLException: ERROR: cached plan must not change result type</code></pre></figure>
<p>(Actually, I first got just the German<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> version of that, which did not help either.)</p>
<p>The funny thing was that this error only happened when I ran the full test suite, and it would only happen after about 15 minutes. So much for rapid trial and error.</p>
<p>After testing out various settings (and different versions of JPA dialects, JDBC drivers, Hibernate versions…), my last guess was that it is not a Spring/JPA/Hibernate-related problem. While each change slightly altered the results (i.e. the number of errors or in which test it would occur), none of the changes really helped.</p>
<p>Finally, after some more searching for <em>just</em> the error message, this <a href="https://github.com/molgenis/molgenis/issues/5511#issuecomment-259893998">Github comment</a> (unrelated issue, unrelated project) gave me my Eureka moment: simply appending <code class="language-plaintext highlighter-rouge">?autosave=ALWAYS</code> to the JDBC URI was all it needed.</p>
<p>In hindsight, there is propbably some side-effect involved in the preceding database tests running in the suite (they also cover some failure scenarios), and this led to this strange issue. Further <a href="https://github.com/pgjdbc/pgjdbc/blob/43e6505e3aa16e6acdf08f02f2dd1e3cf131ac3e/pgjdbc/src/main/java/org/postgresql/PGProperty.java#L383">digging into the code of the Postgres JDBC driver</a> explains what actually happens, and what the options are.</p>
<p>Morale of the story? Append <code class="language-plaintext highlighter-rouge">?autosave=ALWAYS</code> to your JDBC URI if you are running into the same problem! ;-)<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup></p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>While I like the idea of translating technical error messages, a translated message is rather useless without a <em>language-invariant</em> error ID: users need to <em>search</em> for this term! <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>And, also, with a more complex system that has many potential culprits (i.e. debugging ‘targets’), always track back to the question <em>Which part of the setup is most likely the issue?</em> – I got that wrong for quite some time, which slowed down the debugging a lot. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>I had quite some trouble this weekend to get a database integration test suite running with current versions of Spring Boot (1.5.1) and PostgreSQL (9.6.1).OpenVPN, Ubuntu 16.04, and WiFi connection problems2017-01-08T00:00:00+00:002017-01-08T00:00:00+00:00https://roland-ewald.github.io/2017/01/08/openvpn-ubuntu<p>I spent quite some time over the last few weeks to debug a strange issue that happened when connecting to a (newly configured) OpenVPN server with Ubuntu 16.04. Every few minutes the VPN connection simply dropped, with this log message:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">Inactivity timeout (--ping-restart), restarting</code></pre></figure>
<p>The problem occurred sporadically, and nothing (firmware update on router, ‘polling’ a server inside the private network to generate traffic, RTFM, etc.) helped.</p>
<p>Yesterday I finally found a workaround in <a href="http://askubuntu.com/a/689492/284477">this AskUbuntu answer</a> for a similar connection problem: after editing the WiFi connection and setting the IPv6 settings to <code class="language-plaintext highlighter-rouge">Ignore</code>, everything seems to work fine (for now).
Since there are <a href="https://silverdrs.wordpress.com/2015/12/01/openvpn-inactivity-timeout-ping-restart-restarting/">other workarounds</a> for problems with the same symptoms, I suggest trying this first, since it is a simple and quick workaround.</p>
<p>OpenVPN support on Ubuntu can be rather challenging<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> to set up, depending on the server configuration.</p>
<p>Here is what I learned about that in the last weeks:</p>
<ul>
<li>
<p>Depending on your configuration file, <a href="http://askubuntu.com/q/760345/284477">import into the Ubuntu network manager may not work</a>, so you should check if there is a known <a href="https://bugs.launchpad.net/ubuntu/+source/network-manager-openvpn/+bugs">bug</a> that affects you, e.g. missing support for certain OpenVPN options.</p>
</li>
<li>
<p>Even when you got the import working, the network manager may simply not support all settings you need. In this case, you will have to run OpenVPN manually: <code class="language-plaintext highlighter-rouge">sudo openvpn path/to/my/config.ovpn</code> (this is also useful for debugging).</p>
</li>
<li>
<p><a href="https://silverdrs.wordpress.com/2015/12/01/openvpn-inactivity-timeout-ping-restart-restarting/">Some issues</a> are caused by multiple machines being connected to the same OpenVPN server with the same certificate. Only use a single session to rule out any problems in that regard.</p>
</li>
<li>
<p>WiFi can cause a lot of trouble for OpenVPN and you may not notice very brief connection interruptions yourself, so improve your router configuration if you suspect this to be a problem (e.g. switch off the 5GHz band, only use 802.11b+g, etc.).</p>
</li>
<li>
<p>DNS setup also needs an extra step that may be missing from your OpenVPN config file. It <a href="http://askubuntu.com/a/845542/284477">should contain</a>:</p>
</li>
</ul>
<figure class="highlight"><pre><code class="language-text" data-lang="text">script-security 2
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf</code></pre></figure>
<ul>
<li>For more troubleshooting, change the verbosity level in your config file (e.g. to <code class="language-plaintext highlighter-rouge">verb 4</code>) and try again.</li>
</ul>
<p>Good luck!</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>When I configured my clients for the previous VPN setup, it was the other way around: a nightmare on Windows, but a piece of cake on Ubuntu. I guess it really <em>does</em> depend on the specific OpenVPN setup. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>I spent quite some time over the last few weeks to debug a strange issue that happened when connecting to a (newly configured) OpenVPN server with Ubuntu 16.04. Every few minutes the VPN connection simply dropped, with this log message:Let’s try this again…2016-12-25T00:00:00+00:002016-12-25T00:00:00+00:00https://roland-ewald.github.io/2016/12/25/lets-try-again<p>My <a href="/2014/06/09/website-bibtex.html">previous setup</a> was nice in principle, but still required too much overhead regarding the CGI setup, how to ‘freeze’ the page, and so on. The problem was not so much that it was complicated in principle, but it was too complicated to simply change or add some content in five minutes every few weeks, without using any of the tools involved in the meantime (and thus having to go back to the README again and again).</p>
<p>Maybe that was a reason I did not blog as (in)frequently as I would like to have done (and, yes, OK, maybe it is also because I was <a href="https://www.limbus-medtec.com">occupied</a> <a href="https://www.varvis.com">with</a> <a href="https://www.allexes.com">other</a> <a href="https://www.varvis.com/feeder">things</a> ;-).</p>
<p>Nevertheless, I now switched everything to <a href="http://jekyllrb.com">Jekyll</a> and host the site on Github. Even the vanilla setup covers 95% of my use case, and I will live without the other 5%<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>.</p>
<p>Let’s see how that goes. At least I’ve dealt with that particular ‘technical debt’ before the year ends.</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>As a bonus, the site is now served via HTTPS. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>My previous setup was nice in principle, but still required too much overhead regarding the CGI setup, how to ‘freeze’ the page, and so on. The problem was not so much that it was complicated in principle, but it was too complicated to simply change or add some content in five minutes every few weeks, without using any of the tools involved in the meantime (and thus having to go back to the README again and again).