Yannick Pereira-ReisJekyll2020-04-17T18:46:07+00:00https://ypereirareis.github.io/Yannick Pereira-Reishttps://ypereirareis.github.io/https://ypereirareis.github.io/blog/2020/04/09/running-cron-jobs-docker-container-definitive-guide2020-04-09T00:00:00+00:002020-04-09T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p><img src="/images/posts/crontab.png" alt="Crontab" /></p>
<h1 id="tldr-my-cron-jobs-are-not-running-in-my-docker-container-i-dont-know-why-and-its-impossible-to-find-any-logs-anywhere">TLDR: My cron jobs are not running in my docker container. I don’t know why and it’s impossible to find any logs anywhere.</h1>
<ul>
<li>Install syslog/rsyslog into your container and configure it to allow cron logs.</li>
<li>Inspect your syslog logs to find out almost all problems.</li>
<li>Inspect system mail to fix problems left.</li>
</ul>
<h1 id="1-install-syslog-to-have-errors-and-logs-in-the-docker-container">1. Install syslog to have errors and logs in the docker container</h1>
<p>If you run cron daemon in a container, you probably do not have syslog installed and properly configured to have CRON logs and errors.
And if your cron jobs are not running as intended, there is no easy way to find out problems without syslog.</p>
<p>Maybe you only want to install rsyslog to debug your crontab, and not have it in your production docker image and containers.
If so, you can install it in a running container to debug. Then remove/start the container once everything is working.</p>
<h3 id="install-rsyslog">Install rsyslog</h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">apt-get update <span class="o">&&</span> apt-get <span class="nb">install</span> <span class="nt">-y</span> rsyslog</code></pre></figure>
<h3 id="configure-rsyslog-for-cron">Configure rsyslog for cron</h3>
<ul>
<li>Edit <code class="language-plaintext highlighter-rouge">/etc/syslog.conf</code> to enable cron logging (uncomment correct line).</li>
<li>Default cron log file is <code class="language-plaintext highlighter-rouge">/var/log/cron.log</code>.</li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">cron.<span class="k">*</span> /var/log/cron.log</code></pre></figure>
<h3 id="start-rsyslog-and-check-its-running-in-the-container">Start rsyslog and check it’s running in the container</h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span>/etc/init.d/rsyslog restart
<span class="o">[</span> ok <span class="o">]</span> Stopping enhanced syslogd: rsyslogd.
<span class="o">[</span> ok <span class="o">]</span> Starting enhanced syslogd: rsyslogdrsyslogd.
<span class="o">></span>/etc/init.d/rsyslog status
<span class="o">[</span> ok <span class="o">]</span> rsyslogd is running.</code></pre></figure>
<h1 id="2-cron-file-name-must-validate-some-rules">2. Cron file name must validate some rules</h1>
<p>There is a very important thing to consider in this page: <a href="https://www.pantz.org/software/cron/croninfo.html">https://www.pantz.org/software/cron/croninfo.html</a></p>
<blockquote>
<p>Files must conform to the same naming convention as used by run-parts: they must consist solely of upper- and lower-case letters, digits, underscores, and hyphens. Like /etc/crontab, the files in the /etc/cron.d directory are monitored for changes.</p>
</blockquote>
<p><strong>So, if you use to use meaningful extension for your files… forget it for cron files.</strong></p>
<ul>
<li>my-super-cron.cron <strong>is NOT</strong> valid (because of the <strong>dot</strong>).</li>
<li>my-super-cron <strong>is</strong> valid.</li>
<li>my_super_cron <strong>is</strong> valid.</li>
<li>my_SUPER_cRon <strong>is</strong> valid.</li>
<li>my_SUPER_cRon_2020 <strong>is</strong> valid.</li>
</ul>
<p><em>Read the entire link because you will find a lot of useful information about cron in general.</em></p>
<h1 id="3-the-cron-file-owner-must-be-valid-and-properly-defined-to-launch-jobs">3. The cron file owner must be valid and properly defined to launch jobs</h1>
<p>You have mainly two choices where to save your cron files:</p>
<ul>
<li>in the “current” user crontab.</li>
<li>in the global cron location <code class="language-plaintext highlighter-rouge">/etc/cron.d</code> (<code class="language-plaintext highlighter-rouge">/etc/cron.daily,...</code>)</li>
</ul>
<blockquote>
<p>The cron files in /etc/cron.d are a little different than a user’s crontab such that you can specify what user a job runs as.</p>
</blockquote>
<p>If you choose the “current” user crontab, nothing particular to say. But if you choose the <code class="language-plaintext highlighter-rouge">/etc/cron.d</code> location,
your files <strong>MUST BE OWNED</strong> by root. Or you will have “WRONG FILE OWNER” error in your syslog.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">Apr 1 19:35:01 node1 cron[32]: <span class="o">(</span><span class="k">*</span>system<span class="k">*</span>my-super-cron<span class="o">)</span> WRONG FILE OWNER <span class="o">(</span>/etc/cron.d/my-super-cron<span class="o">)</span></code></pre></figure>
<p>The file MUST BE owned by root, to avoid security hole such as privilege escalation, for instance:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="k">*</span> <span class="k">*</span> <span class="k">*</span> <span class="k">*</span> <span class="k">*</span> root <span class="nb">cp</span> /bin/bash /tmp/nowimroot <span class="o">&&</span> <span class="nb">chown </span>root:root /tmp/nowimroot <span class="o">&&</span> <span class="nb">chmod </span>u+s /tmp/nowimroot</code></pre></figure>
<h1 id="4-new-line-before-eof-end-of-file-is-mandatory">4. New line before EOF (end of file) is mandatory</h1>
<p>If you are, for example, a developer, you probably are aware of that recommendation to always add an new line at the end of a file.
But if you wonder <strong>WHY?</strong>, just read this thread and the very good explanation:</p>
<p><a href="https://stackoverflow.com/questions/729692/why-should-text-files-end-with-a-newline#answer-729795">https://stackoverflow.com/questions/729692/why-should-text-files-end-with-a-newline#answer-729795</a></p>
<p>And for your cron files, you must follow this rule and add a new line at the end of files or you will have this error in syslog:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">Apr 1 20:04:01 node1 cron[32]: <span class="o">(</span><span class="k">*</span>system<span class="k">*</span>my-super-cron<span class="o">)</span> ERROR <span class="o">(</span>Missing newline before EOF, this crontab file will be ignored<span class="o">)</span></code></pre></figure>
<p>You can easily test the difference between two files using the following commands:</p>
<ul>
<li>to create an invalid file:</li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">echo</span> <span class="nt">-n</span> <span class="s2">"* * * * * root echo </span><span class="se">\"</span><span class="s2">test</span><span class="se">\"</span><span class="s2"> > /tmp/test-cron-invalid.log"</span> <span class="o">></span> /etc/cron.d/invalid-cron</code></pre></figure>
<ul>
<li>to create a valid file:</li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="sb">`</span><span class="nb">echo</span> <span class="s2">"* * * * * root echo </span><span class="se">\"</span><span class="s2">test</span><span class="se">\"</span><span class="s2"> > /tmp/test-cron-valid.log"</span> <span class="o">></span> /etc/cron.d/valid-cron<span class="sb">`</span></code></pre></figure>
<p>With <code class="language-plaintext highlighter-rouge">echo -n</code> you will create a crontab line without new line (CRLF), so it will fail with previous error in syslog.</p>
<h1 id="5-environment-variables-are-not-defined-or-available-as-expected-in-crontab">5. Environment variables are not defined or available as expected in crontab</h1>
<blockquote>
<p>When you use cron, your env is not same as if you log in. Depends which *nix system you have.</p>
</blockquote>
<blockquote>
<p>That is your env when you use cron. It’s not same as login. PATH is something, not enough usually and so on. Usually HOME is.</p>
</blockquote>
<h3 id="the-cron-daemon-automatically-sets-several-environment-variables">The cron daemon automatically sets several environment variables.</h3>
<ul>
<li>The default path is set to PATH=/usr/bin:/bin. If the command you are executing is not present in the cron specified path, you can either use the absolute path to the command or change the cron $PATH variable. You can’t implicitly append :$PATH as you would do with a regular script.</li>
<li>The default shell is set to /bin/sh. To change the different shell, use the SHELL variable.</li>
<li>Cron invokes the command from the user’s home directory. The HOME variable can be set in the crontab.</li>
<li>The email notification is sent to the owner of the crontab. To overwrite the default behavior, you can use the MAILTO environment variable with a list (comma separated) of all the email addresses you want to receive the email notifications. When MAILTO is defined but empty (MAILTO=””), no mail is sent.</li>
</ul>
<h3 id="consider-all-your-user-or-custom-defined-env-variables-are-not-available-in-crontab">Consider all your user (or custom) defined env variables are not available in crontab.</h3>
<p>I can give you 3 solutions to have your variables available and used in your cron:</p>
<ul>
<li><strong>In the cron file itself</strong></li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Env</span>
<span class="nv">SHELL</span><span class="o">=</span>/bin/bash
<span class="nv">HOME</span><span class="o">=</span>/home/sandbox
<span class="nv">PATH</span><span class="o">=</span>/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin
<span class="nv">APP_DATABASE_HOST</span><span class="o">=</span>localhost
<span class="c"># Cron</span>
<span class="k">*</span> <span class="k">*</span> <span class="k">*</span> <span class="k">*</span> <span class="k">*</span> root <span class="nb">echo</span> <span class="s2">"test"</span> <span class="o">></span> /tmp/test-cron-valid.log</code></pre></figure>
<ul>
<li><strong>In a custom script sourced in the crontab line.</strong></li>
</ul>
<p>I wrote an article about that few years ago:
<a href="/blog/2016/02/29/docker-crontab-environment-variables/">Access env variables from crontab into a container</a><br />
<strong>Résumé:</strong> dynamically build a script that export env variables, and source this file in your crontab line.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="k">*</span> <span class="k">*</span> <span class="k">*</span> <span class="k">*</span> <span class="k">*</span> root <span class="nb">source </span>custom_vars.sh <span class="o">&&</span> <span class="nb">echo</span> <span class="s2">"test"</span> <span class="o">></span> /tmp/test-cron-valid.log</code></pre></figure>
<ul>
<li><strong>In /etc/environment</strong></li>
</ul>
<p>If you want to know where cron daemon is reading system variables you can look in file <code class="language-plaintext highlighter-rouge">/etc/pam.d/cron</code>.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">cat</span> /etc/pam.d/cron
<span class="c"># The PAM configuration file for the cron daemon</span>
@include common-auth
<span class="c"># Sets the loginuid process attribute</span>
session required pam_loginuid.so
<span class="c"># Read environment variables from pam_env's default files, /etc/environment</span>
<span class="c"># and /etc/security/pam_env.conf.</span>
session required pam_env.so
<span class="c"># In addition, read system locale information</span>
session required pam_env.so <span class="nv">envfile</span><span class="o">=</span>/etc/default/locale
@include common-account
@include common-session-noninteractive
<span class="c"># Sets up user limits, please define limits for cron tasks</span>
<span class="c"># through /etc/security/limits.conf</span>
session required pam_limits.so</code></pre></figure>
<p>In my configuration, cron daemon loads environment variables from <code class="language-plaintext highlighter-rouge">/etc/environment</code>.
So, we can add variables we want to use in cron, in this file. Either statically or dynamically.
<strong>But all variables will be available in all cron jobs.</strong></p>
<p>As a simple example, append user variables prefixed by <strong>CUSTOM_</strong> into <code class="language-plaintext highlighter-rouge">/etc/environment</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">printenv</span> | <span class="nb">grep</span> <span class="nt">-E</span> <span class="s2">"^CUSTOM_"</span> <span class="o">>></span> /etc/environment</code></pre></figure>
<h1 id="6-crontab-file-syntax-errors-permission-denied-example">6. Crontab file syntax errors (Permission denied example)</h1>
<p>It’s possible you have avoided all the previous pitfalls. But there is a few more things to worry about:
<strong>syntax or permission errors in the crontab line itself.</strong></p>
<p>Let’s take this example:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="k">*</span> <span class="k">*</span> <span class="k">*</span> <span class="k">*</span> <span class="k">*</span> www-data <span class="nb">echo</span> <span class="s2">"test"</span> <span class="o">>></span> /var/log/test-cron.log</code></pre></figure>
<ul>
<li>Every minute of every hour, of every…</li>
<li>as the <strong>www-data</strong> user…</li>
<li>we (try to) append the word “test”…</li>
<li>in the file <code class="language-plaintext highlighter-rouge">/var/log/test-cron.log</code></li>
</ul>
<p><strong>But…</strong> the log file <code class="language-plaintext highlighter-rouge">/var/log/test-cron.log</code> is never created and there is no error in syslog.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span><span class="nb">tail</span> <span class="nt">-50f</span> /var/log/cron.log
Apr 8 19:31:01 node1 CRON[30034]: <span class="o">(</span>www-data<span class="o">)</span> CMD <span class="o">(</span><span class="nb">echo</span> <span class="s2">"test"</span> <span class="o">></span> /var/log/test-cron.log<span class="o">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span><span class="nb">cat</span> /var/log/test-cron.log
<span class="nb">cat</span>: /var/log/test-cron.log: No such file or directory</code></pre></figure>
<p>Actually, when there is an error in the crontab line (Syntax, permission,…), cron daemon sends an email with the error.</p>
<ul>
<li>If you have configured everything correctly to send emails from your server, you will have to look in your email box.</li>
<li>If not, just have a look in the <code class="language-plaintext highlighter-rouge">/var/spool/mail/{USER}</code> file, where emails are stored.
The USER is the one defined in the crontab line. “www-data” in the example.</li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span><span class="nb">cat</span> /var/spool/mail/www-data
From www-data@node1 Wed Apr 08 19:33:01 2020
Return-path: <www-data@node1>
Envelope-to: www-data@node1
Delivery-date: Wed, 08 Apr 2020 19:33:01 +0200
Received: from www-data by node1
<span class="o">(</span>envelope-from <www-data@node1><span class="o">)</span>
<span class="nb">id </span>qdlaAZel-Asd54-ddlzt
<span class="k">for </span>www-data@node1<span class="p">;</span> Wed, 08 Apr 2020 19:33:01 +0200
From: root@node1 <span class="o">(</span>Cron Daemon<span class="o">)</span>
To: www-data@node1
Subject: Cron <www-data@node1> <span class="nb">echo</span> <span class="s2">"test"</span> <span class="o">></span> /var/log/test-cron.log
MIME-Version: 1.0
Content-Type: text/plain<span class="p">;</span> <span class="nv">charset</span><span class="o">=</span>US-ASCII
Content-Transfer-Encoding: 8bit
X-Cron-Env: <<span class="nv">SHELL</span><span class="o">=</span>/bin/sh>
X-Cron-Env: <<span class="nv">HOME</span><span class="o">=</span>/var/www/home>
X-Cron-Env: <<span class="nv">PATH</span><span class="o">=</span>/usr/bin:/bin>
X-Cron-Env: <<span class="nv">LOGNAME</span><span class="o">=</span>www-data>
Message-Id: <qdlaAZel-Asd54-ddlzt@node1>
Date: Wed, 08 Apr 2020 19:33:01 +0200
/bin/sh: 1: cannot create /var/log/test-cron.log: Permission denied</code></pre></figure>
<p><strong>Tadaaam!!</strong></p>
<h1 id="resources">Resources</h1>
<ul>
<li>My personal experience with docker and cron</li>
<li><a href="https://www.pantz.org/software/cron/croninfo.html">https://www.pantz.org/software/cron/croninfo.html</a></li>
<li><a href="https://stackoverflow.com/questions/729692/why-should-text-files-end-with-a-newline#answer-729795">https://stackoverflow.com/questions/729692/why-should-text-files-end-with-a-newline#answer-729795</a></li>
<li><a href="https://serverfault.com/questions/566437/cron-task-error-wrong-file-owner#answer-566442">https://serverfault.com/questions/566437/cron-task-error-wrong-file-owner#answer-566442</a></li>
<li><a href="https://linuxize.com/post/scheduling-cron-jobs-with-crontab/">https://linuxize.com/post/scheduling-cron-jobs-with-crontab/</a></li>
<li><a href="https://www.unix.com/shell-programming-and-scripting/163494-setting-environment-variables-cron-file.html">https://www.unix.com/shell-programming-and-scripting/163494-setting-environment-variables-cron-file.html</a></li>
</ul>
<p><a href="https://ypereirareis.github.io/blog/2020/04/09/running-cron-jobs-docker-container-definitive-guide/">Definitive guide on how to setup up and running cron jobs in docker containers</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on April 09, 2020.</p>
https://ypereirareis.github.io/blog/2020/02/21/how-to-join-prometheus-metrics-by-label-with-promql2020-02-21T00:00:00+00:002020-02-21T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p><img src="/images/posts/prometheus.jpg" alt="Prometheus" /></p>
<h1 id="tldr-questions-answered-in-this-article">TLDR: questions answered in this article.</h1>
<ul>
<li>How to JOIN two different Prometheus metrics by label with PromQL.</li>
</ul>
<h1 id="available-metrics-for-the-example">Available metrics for the example</h1>
<p>Let’s say we use the excellent “node-exporter” project to monitor our servers.</p>
<ul>
<li><a href="https://github.com/prometheus/node_exporter">https://github.com/prometheus/node_exporter</a></li>
<li><a href="https://hub.docker.com/r/prom/node-exporter/">https://hub.docker.com/r/prom/node-exporter/</a></li>
</ul>
<p><strong>We will have metrics looking like that, for example:</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>node_disk_bytes_read<span class="o">{}</span>
node_disk_bytes_read<span class="o">{</span><span class="nv">device</span><span class="o">=</span><span class="s2">"dm-0"</span>,instance<span class="o">=</span><span class="s2">"10.0.0.10"</span>,job<span class="o">=</span><span class="s2">"node-exporter"</span><span class="o">}</span> | 43161334784
</code></pre></div></div>
<p>We can use PromQL to build aggregations, sum of “disk bytes read” by instance/server:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sum</span><span class="o">(</span>node_disk_bytes_read<span class="o">{})</span> by <span class="o">(</span>instance<span class="o">)</span>
<span class="o">{</span><span class="nv">instance</span><span class="o">=</span><span class="s2">"10.0.0.8"</span><span class="o">}</span> | 22082332072448
<span class="o">{</span><span class="nv">instance</span><span class="o">=</span><span class="s2">"10.0.0.9"</span><span class="o">}</span> | 8439202548224
<span class="o">{</span><span class="nv">instance</span><span class="o">=</span><span class="s2">"10.0.0.10"</span><span class="o">}</span> | 28203612148224
<span class="o">{</span><span class="nv">instance</span><span class="o">=</span><span class="s2">"10.0.0.11"</span><span class="o">}</span> | 56887513977344
<span class="o">{</span><span class="nv">instance</span><span class="o">=</span><span class="s2">"10.0.0.12"</span><span class="o">}</span> | 30887053824
<span class="o">{</span><span class="nv">instance</span><span class="o">=</span><span class="s2">"10.0.0.13"</span><span class="o">}</span> | 36352176166912
</code></pre></div></div>
<p>With our custom usage of node-exporter we have added a custom metric called “node_meta”.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/sh -e</span>
<span class="nv">NODE_NAME</span><span class="o">=</span><span class="si">$(</span><span class="nb">cat</span> /etc/nodename<span class="si">)</span>
<span class="nb">echo</span> <span class="s2">"node_meta{node_id=</span><span class="se">\"</span><span class="nv">$NODE_ID</span><span class="se">\"</span><span class="s2">, container_label_com_docker_swarm_node_id=</span><span class="se">\"</span><span class="nv">$NODE_ID</span><span class="se">\"</span><span class="s2">, node_name=</span><span class="se">\"</span><span class="nv">$NODE_NAME</span><span class="se">\"</span><span class="s2">} 1"</span> <span class="o">></span> /etc/node-exporter/node-meta.prom
<span class="nb">set</span> <span class="nt">--</span> /bin/node_exporter <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="nb">exec</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
</code></pre></div></div>
<ul>
<li>You can see this configuration here <a href="https://github.com/stefanprodan/swarmprom/blob/master/node-exporter/conf/docker-entrypoint.sh">https://github.com/stefanprodan/swarmprom/blob/master/node-exporter/conf/docker-entrypoint.sh</a></li>
<li>The full project is available here <a href="https://github.com/stefanprodan/swarmprom">https://github.com/stefanprodan/swarmprom</a></li>
</ul>
<p><strong>With this configuration we also have metrics like:</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>node_meta<span class="o">{}</span>
</code></pre></div></div>
<p>We can query Prometheus to have values for this metric:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>node_meta<span class="o">{}</span>
node_meta<span class="o">{</span><span class="nv">instance</span><span class="o">=</span><span class="s2">"10.0.0.8"</span>,job<span class="o">=</span><span class="s2">"node-exporter"</span>,node_name<span class="o">=</span><span class="s2">"node2"</span><span class="o">}</span> | 1
node_meta<span class="o">{</span><span class="nv">instance</span><span class="o">=</span><span class="s2">"10.0.0.9"</span>,job<span class="o">=</span><span class="s2">"node-exporter"</span>,node_name<span class="o">=</span><span class="s2">"node4"</span><span class="o">}</span> | 1
node_meta<span class="o">{</span><span class="nv">instance</span><span class="o">=</span><span class="s2">"10.0.0.10"</span>,job<span class="o">=</span><span class="s2">"node-exporter"</span>,node_name<span class="o">=</span><span class="s2">"node5"</span><span class="o">}</span> | 1
node_meta<span class="o">{</span><span class="nv">instance</span><span class="o">=</span><span class="s2">"10.0.0.11"</span>,job<span class="o">=</span><span class="s2">"node-exporter"</span>,node_name<span class="o">=</span><span class="s2">"node6"</span><span class="o">}</span> | 1
node_meta<span class="o">{</span><span class="nv">instance</span><span class="o">=</span><span class="s2">"10.0.0.12"</span>,job<span class="o">=</span><span class="s2">"node-exporter"</span>,node_name<span class="o">=</span><span class="s2">"node3"</span><span class="o">}</span> | 1
node_meta<span class="o">{</span><span class="nv">instance</span><span class="o">=</span><span class="s2">"10.0.0.13"</span>,job<span class="o">=</span><span class="s2">"node-exporter"</span>,node_name<span class="o">=</span><span class="s2">"node1"</span><span class="o">}</span> | 1
</code></pre></div></div>
<ul>
<li>You can notice that here we have labels allowing us to have a match between an <strong>instance IP address (10.0.0.8)</strong> and an <strong>instance name (node2)</strong>.</li>
<li>There is a label in common between the two metrics “node_meta” and “node_disk_bytes_read”: <strong>instance</strong>.</li>
</ul>
<p><strong>QUESTION?</strong></p>
<p>How to query prometheus to have <strong>sum of “disk bytes read”</strong> by instance/node/server name ? The result we want is something like that :</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>node2 <span class="o">=></span> 22082332072448
node4 <span class="o">=></span> 8439202548224
node5 <span class="o">=></span> 28203612148224
node6 <span class="o">=></span> 56887513977344
node3 <span class="o">=></span> 30887053824
node1 <span class="o">=></span> 36352176166912
</code></pre></div></div>
<h1 id="how-to-join-the-metrics">How to JOIN the metrics</h1>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sum</span><span class="o">(</span>node_disk_bytes_read <span class="k">*</span> on<span class="o">(</span>instance<span class="o">)</span> group_left<span class="o">(</span>node_name<span class="o">)</span> node_meta<span class="o">{})</span> by <span class="o">(</span>node_name<span class="o">)</span>
</code></pre></div></div>
<ul>
<li><strong>on(instance)</strong> => this is how to JOIN on label <strong>instance</strong>.</li>
<li><strong>group_left(node_name) node_meta{}</strong> => means, keep the label <strong>node_name</strong> from metric <strong>node_meta</strong> in the result.</li>
</ul>
<p>And the result is:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">{</span><span class="nv">node_name</span><span class="o">=</span><span class="s2">"node2"</span><span class="o">}</span> | 22082332072448
<span class="o">{</span><span class="nv">node_name</span><span class="o">=</span><span class="s2">"node4"</span><span class="o">}</span> | 8439202548224
<span class="o">{</span><span class="nv">node_name</span><span class="o">=</span><span class="s2">"node5"</span><span class="o">}</span> | 28203612148224
<span class="o">{</span><span class="nv">node_name</span><span class="o">=</span><span class="s2">"node6"</span><span class="o">}</span> | 56887513977344
<span class="o">{</span><span class="nv">node_name</span><span class="o">=</span><span class="s2">"node3"</span><span class="o">}</span> | 30887053824
<span class="o">{</span><span class="nv">node_name</span><span class="o">=</span><span class="s2">"node1"</span><span class="o">}</span> | 36352176166912
</code></pre></div></div>
<p><strong>Tadaaam!!</strong></p>
<p><a href="https://ypereirareis.github.io/blog/2020/02/21/how-to-join-prometheus-metrics-by-label-with-promql/">How to join Prometheus metrics by label with PromQL</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on February 21, 2020.</p>
https://ypereirareis.github.io/blog/2020/02/18/how-to-reduce-nginx-502-bad-gateway-errors-risks-with-dynamic-domain-name-resolution2020-02-18T00:00:00+00:002020-02-18T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p><img src="/images/posts/nginx.gif" alt="PHP-FPM" /></p>
<h1 id="tldr-questions-answered-in-this-article">TLDR: questions answered in this article.</h1>
<ul>
<li>How to avoid Nginx reload on php-fpm restart ?</li>
<li>How to reduce 502 Bad gateway errors with Nginx and php-fpm ?</li>
<li>How to configure dynamic domain name (DNS) resolution in Nginx ?</li>
</ul>
<h1 id="nginx-common-errors-leading-to-502-bad-gateway">Nginx common errors leading to 502 Bad Gateway</h1>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>error] 12#0: <span class="k">*</span>16 connect<span class="o">()</span> failed <span class="o">(</span>111: Connection refused<span class="o">)</span> <span class="k">while </span>connecting to upstream
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>error] 12#0: <span class="k">*</span>20 php could not be resolved <span class="o">(</span>2: Server failure<span class="o">)</span>
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>error] 12#0: <span class="k">*</span>36 connect<span class="o">()</span> failed <span class="o">(</span>113: No route to host<span class="o">)</span> <span class="k">while </span>connecting to upstream
</code></pre></div></div>
<h1 id="a-common-php-upstream-configuration-generating-problems">A common PHP upstream configuration generating problems</h1>
<p><strong>/etc/nginx/conf.d/upstream.conf</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>upstream php-upstream <span class="o">{</span> server php:9000<span class="p">;</span> <span class="o">}</span>
</code></pre></div></div>
<p><strong>/etc/nginx/sites-enabled/app.conf</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>server <span class="o">{</span>
root /var/www/html/web<span class="p">;</span>
listen <span class="k">*</span>:80<span class="p">;</span>
location ~ ^/index<span class="se">\.</span>php<span class="o">(</span>/|<span class="nv">$)</span> <span class="o">{</span>
fastcgi_pass php-upstream<span class="p">;</span>
...
internal<span class="p">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>With that kind of configuration the domain name resolution for “php” is done only once when Nginx is started or reloaded.
So any problem with php-fpm going down for instance, will need a restart or reload for Nginx.
<strong>It’s not a very resilient configuration.</strong></p>
<p>Let’s say we are running PHP-FPM in a docker container
and our container is restarted by the orchestrator, or we deploy a new version of our stack not including any Nginx changes we will need to restart or reload Nginx too or we will have a lot of 502 Bad Gateway errors.
<strong>To avoid this we need to use dynamic domain name resolution !!!</strong></p>
<h1 id="dynamic-domain-name-resolution-configuration-for-nginx">Dynamic domain name resolution configuration for Nginx</h1>
<h2 id="resources">Resources</h2>
<ul>
<li><a href="https://github.com/rancher/rancher/issues/7691#issuecomment-277635645">https://github.com/rancher/rancher/issues/7691#issuecomment-277635645</a></li>
<li><a href="https://stackoverflow.com/questions/35744650/docker-network-nginx-resolver#answer-37656784">https://stackoverflow.com/questions/35744650/docker-network-nginx-resolver#answer-37656784</a></li>
<li><a href="https://www.nginx.com/blog/dns-service-discovery-nginx-plus/#Methods-for-Service-Discovery-with-DNS-for-NGINX-and-NGINX Plus">https://www.nginx.com/blog/dns-service-discovery-nginx-plus/#Methods-for-Service-Discovery-with-DNS-for-NGINX-and-NGINX Plus</a></li>
</ul>
<p><strong>No more upstream configuration needed !!!</strong></p>
<p>And the application configuration file becomes:</p>
<p><strong>/etc/nginx/sites-enabled/app.conf</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>server <span class="o">{</span>
root /var/www/html/web<span class="p">;</span>
listen <span class="k">*</span>:80<span class="p">;</span>
location ~ ^/index<span class="se">\.</span>php<span class="o">(</span>/|<span class="nv">$)</span> <span class="o">{</span>
resolver 127.0.0.11 <span class="nv">valid</span><span class="o">=</span>10s <span class="nv">ipv6</span><span class="o">=</span>off<span class="p">;</span>
<span class="nb">set</span> <span class="nv">$backendfpm</span> <span class="s2">"php:9000"</span><span class="p">;</span>
fastcgi_pass <span class="nv">$backendfpm</span><span class="p">;</span>
...
internal<span class="p">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<ul>
<li><strong>127.0.0.11</strong> is the internal docker DNS address name.</li>
<li><strong>valid=10s</strong> because we want to re‑resolve names every 10 seconds.</li>
<li><strong>ipv6=off</strong> because we do not want to use IPv6.</li>
<li><strong>fastcgi_pass $backendfpm;</strong> would be the same for proxy_pass.</li>
</ul>
<p>When you use a variable to specify the domain name in the proxy_pass directive,
NGINX re‑resolves the domain name when its TTL expires.
You must include the resolver directive to explicitly specify the name server (NGINX does not refer to /etc/resolv.conf as in the first two methods).
By including the valid parameter to the resolver directive, you can tell NGINX to ignore the TTL and re‑resolve names at a specified frequency instead.
Here we tell NGINX to re‑resolve names every 10 seconds.</p>
<p>This method eliminates two drawbacks of the first method,
in that the NGINX startup or reload operation doesn’t fail when the domain name can’t be resolved,
and we can control how often NGINX re‑resolves the name.</p>
<h1 id="docker-swarm-and-kubernetes">Docker Swarm and Kubernetes</h1>
<p>It’s a common problem with docker Swarm and Kubernetes as everything is containerized and a lot of networking stuffs comes into game.</p>
<h2 id="docker-swarm-services">Docker swarm services</h2>
<p><strong>Let’s take this simple docker Swarm configuration</strong></p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">3.5'</span>
<span class="na">services</span><span class="pi">:</span>
<span class="na">php</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">private.docker-registry.tld/php</span>
<span class="na">networks</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">default</span>
<span class="na">nginx</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">private.docker-registry.tld/nginx</span>
<span class="na">networks</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">default</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">80:80</span>
<span class="na">networks</span><span class="pi">:</span>
<span class="na">default</span><span class="pi">:</span>
</code></pre></div></div>
<ul>
<li>Then start this docker swarm stack</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker stack deploy <span class="nt">-c</span> docker-compose.yml <span class="nt">--with-registry-auth</span> my-super-project
</code></pre></div></div>
<ul>
<li>You will have 2 services starting one for nginx, one for php</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">></span>docker service <span class="nb">ls</span> <span class="nt">--format</span><span class="o">=</span><span class="s2">"{{.Name}}"</span> | <span class="nb">grep </span>my-super-project
my-super-project_nginx
my-super-project_php
</code></pre></div></div>
<ul>
<li>You will have 1 network created for the entire stack</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">></span>docker network <span class="nb">ls</span> <span class="nt">--format</span><span class="o">=</span><span class="s2">"{{.Name}}"</span> | <span class="nb">grep </span>my-super-project
my-super-project_default
</code></pre></div></div>
<ul>
<li>You will have many containers (some for nginx, some for php)</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">></span>docker stack ps my-super-project <span class="nt">--format</span><span class="o">=</span><span class="s2">"{{.ID }} {{.Node }} {{.Name}}"</span>
v15bn01z2yd3 node01 my-super-project_php.1
v236jgodsm4k node02 my-super-project_nginx.1
7omiwzm4i1wj node02 my-super-project_php.2
g9c55nzn13lr node01 my-super-project_nginx.2
</code></pre></div></div>
<ul>
<li>Then, go into a nginx container and then ping PHP</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">></span>docker <span class="nb">exec</span> <span class="nt">-it</span> my-super-project_nginx.2.g9c55nzn13lr5z0kulegfpu0q bash
root@node01:/# ping php
PING php <span class="o">(</span>10.224.248.16<span class="o">)</span> 56<span class="o">(</span>84<span class="o">)</span> bytes of data.
64 bytes from 10.224.248.16: <span class="nv">icmp_seq</span><span class="o">=</span>1 <span class="nv">ttl</span><span class="o">=</span>64 <span class="nb">time</span><span class="o">=</span>0.353 ms
</code></pre></div></div>
<p><strong>But what is this IP address ?</strong></p>
<ul>
<li>It’s not so obvious to find out the first time. It’s actually the (virtual) IP of the PHP service: <strong>my-super-project_php</strong>.</li>
<li>And this PHP service is associated to the network to act as a load balancer between PHP containers.</li>
<li>And it’s this particular IP that is resolved by Nginx.</li>
</ul>
<p><strong>How to find this IP in the docker networking mess ?</strong></p>
<p>Please note the important <code class="language-plaintext highlighter-rouge">--verbose</code> option here</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker network inspect my-super-project_default <span class="nt">--verbose</span>
</code></pre></div></div>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s2">"</span><span class="s">my-super-project_php"</span><span class="pi">:</span> <span class="pi">{</span>
<span class="s2">"</span><span class="s">VIP"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">10.224.248.16"</span><span class="pi">,</span>
<span class="s2">"</span><span class="s">Ports"</span><span class="pi">:</span> <span class="pi">[],</span>
<span class="s2">"</span><span class="s">LocalLBIndex"</span><span class="pi">:</span> <span class="nv">2646</span><span class="pi">,</span>
<span class="s2">"</span><span class="s">Tasks"</span><span class="pi">:</span> <span class="pi">[</span>
<span class="pi">{</span>
<span class="s2">"</span><span class="s">Name"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">my-super-project_php.1.v15bn01z2yd3q5ftu7aog87pg"</span><span class="pi">,</span>
<span class="nv">...</span>
<span class="pi">},</span>
<span class="pi">{</span>
<span class="s2">"</span><span class="s">Name"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">my-super-project_php.2.7omiwzm4i1wj1whqfuwku6921"</span><span class="pi">,</span>
<span class="nv">...</span>
<span class="pi">}</span>
<span class="pi">]</span>
<span class="pi">}</span><span class="err">,</span>
</code></pre></div></div>
<ul>
<li><strong>AND NOW…</strong>, if you want to have a big beautiful 502 Bad gateway error, just remove this PHP service.</li>
</ul>
<p>No more ways to contact PHP containers.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker service <span class="nb">rm </span>my-super-project_php
</code></pre></div></div>
<ul>
<li>Start this service again and you will see that this vurtual IP has changed, so a re-resolution domain name is required for Nginx</li>
</ul>
<p><strong>Tadaaam!!</strong></p>
<p><a href="https://ypereirareis.github.io/blog/2020/02/18/how-to-reduce-nginx-502-bad-gateway-errors-risks-with-dynamic-domain-name-resolution/">How to reduce Nginx 502 bad gateway errors and risks with dynamic domain name resolution for proxy_pass and fastcgi_pass</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on February 18, 2020.</p>
https://ypereirareis.github.io/blog/2019/10/28/why-you-should-split-env-file-with-docker-compose-docker-swarm-stack-and-services2019-10-28T00:00:00+00:002019-10-28T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p><img src="/images/posts/docker.svg" alt="PHP-FPM" /></p>
<h1 id="tldr-questions-answered-in-this-article">TLDR: questions answered in this article.</h1>
<ul>
<li>How to improve docker .env configuration for security ?</li>
<li>How to improve docker .env configuration for performance ?</li>
<li>How to improve docker .env configuration for better rolling update ?</li>
<li>How to improve docker swarm rolling update time ?</li>
<li>Why all my containers are restarting when updating environnement variables ?</li>
<li>How to properly split environnement files ?</li>
</ul>
<h1 id="using-a-env-file-to-store-configuration-with-docker-compose-swarm-stacks-and-services">Using a <code class="language-plaintext highlighter-rouge">.env</code> file to store configuration with docker-compose swarm stacks and services.</h1>
<p>Since a few years, a lot of projects based on docker, even in the open source community, comes with a <code class="language-plaintext highlighter-rouge">.env</code> file to store configuration.
It allows to define specific configuration for deployments (development, staging and production for instance).
It’s a good starting point if your try to respect the <a href="https://en.wikipedia.org/wiki/Twelve-Factor_App_methodology">twelve-factor app methodology</a>,
but you should NOT use a single <code class="language-plaintext highlighter-rouge">.env</code> file with docker swarm stack and services or you will have some organisation, security and performance problems.</p>
<p>Let’s take a PHP <a href="https://symfony.com/">Symfony</a> project developed and deployed thanks to docker and docker-compose on a swarm cluster.
In that kind of configuration we will often work with a single <code class="language-plaintext highlighter-rouge">.env</code> file used by every part of our technical stack:</p>
<ul>
<li>the <a href="https://docs.docker.com/compose/environment-variables/#the-env-file">docker-compose file</a></li>
<li>the <a href="https://docs.docker.com/compose/environment-variables/#the-env_file-configuration-option">running container</a></li>
<li>the <a href="https://symfony.com/doc/current/configuration.html#configuration-based-on-environment-variables">Symfony app itself</a></li>
</ul>
<p><strong>Let’s take these typical <code class="language-plaintext highlighter-rouge">docker-compose.yml</code> and <code class="language-plaintext highlighter-rouge">.env</code> files examples:</strong></p>
<script src="https://gist.github.com/ypereirareis/2aa1fbc62d31088bf3845d6beb3a109e.js"></script>
<h1 id="what-are-the-problems-in-the-case-of-single-env-file-">What are the problems in the case of single <code class="language-plaintext highlighter-rouge">.env</code> file ?</h1>
<h2 id="problem-1-code-organization-and-separation-of-concerns">Problem #1: Code organization and separation of concerns.</h2>
<ul>
<li>Very difficult to say which environment variables are used by each service.</li>
<li>Very difficult to say which environment variables are used by the PHP application itself.</li>
<li>Difficult to pick up a single service from that stack to add it to another stack.</li>
<li>Configurations can be mixed in this single file (unless you organize each part with comments).</li>
</ul>
<h2 id="problem-2-bugs">Problem #2: Bugs.</h2>
<p>Let’s say you use a project like this perfect one: <a href="https://github.com/jwilder/nginx-proxy">jwilder/nginx-proxy</a>.</p>
<ul>
<li>You will add an env variable <code class="language-plaintext highlighter-rouge">VIRTUAL_HOST=domain.localhost</code> to your <code class="language-plaintext highlighter-rouge">.env</code> file to access Nginx web server with your custom domain “domain.localhost”.</li>
<li>You will access <a href="http://domain.localhost">http://domain.localhost</a> URL and you will have intermittent errors, and the reverse proxy logs will look like:</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nginx.1 | 2019/10/07 07:32:39 <span class="o">[</span>error] 147#147: <span class="k">*</span>105 recv<span class="o">()</span> failed <span class="o">(</span>104: Connection reset by peer<span class="o">)</span> <span class="k">while </span>reading response header from upstream, client: 172.17.0.1, server: domain.localhost, request: <span class="s2">"POST / HTTP/1.1"</span>, upstream: <span class="s2">"http://172.17.0.10:9000"</span>, host: <span class="s2">"domain.localhost"</span>, referrer: <span class="s2">"http://domain.localhost"</span>
nginx.1 | 2019/10/07 07:32:39 <span class="o">[</span>warn] 147#147: <span class="k">*</span>105 upstream server temporarily disabled <span class="k">while </span>reading response header from upstream, client: 172.17.0.1, server: domain.localhost, request: <span class="s2">"POST / HTTP/1.1"</span>, upstream: <span class="s2">"http://172.17.0.10:9000"</span>, host: <span class="s2">"domain.localhost"</span>, referrer: <span class="s2">"http://domain.localhost"</span>
</code></pre></div></div>
<p>This is because adding a variable to the <code class="language-plaintext highlighter-rouge">.env</code> file, will add it into <strong>all containers</strong> (configured with <code class="language-plaintext highlighter-rouge">env_file</code> directive).
So with the <strong>round robin</strong> algorithm used by default within Nginx load balancing, the request will reach the PHP container one time out of two, instead of the Nginx container.</p>
<h2 id="problem-3-security">Problem #3: Security.</h2>
<p>As we said in the previous part, “adding a variable to the <code class="language-plaintext highlighter-rouge">.env</code> file, will add it into <strong>all containers</strong> (configured with <code class="language-plaintext highlighter-rouge">env_file</code> directive)”.
So you will have access all environment variables in all running containers… In addition to the fact that it is unnecessary, it can introduce vulnerabilities.</p>
<p>Imagine your Nginx web server is compromised and some hackers can export all env variables available in your Nginx running container, they will have access to sensitive information like database credentials:</p>
<ul>
<li>DB_HOST=127.0.0.1</li>
<li>DB_PORT=3306</li>
<li>DB_NAME=test</li>
<li>DB_USER=root</li>
<li>DB_PASSWORD=very_sensitive_password</li>
</ul>
<p>They will also have access to other sensitive information:</p>
<ul>
<li>you are running your application from a docker container whose image is stored in a private registry <code class="language-plaintext highlighter-rouge">REGISTRY_PATH</code> (registry.domain.tld)</li>
<li>you are using port number 2000 for your app, so are probably behind a reverse proxy.</li>
<li>you are using PHP for your application, probably version 7.2.</li>
<li>…</li>
</ul>
<h2 id="problem-4-stack-update-performance-and-resources-consumption">Problem #4: Stack update, performance and resources consumption.</h2>
<p>Docker has a built-in mecanism allowing to restart containers when dependencies have changed: env variables, docker-compose directives, networks,…
I’m sure you see where I’m going with this… Imagine you want to increase PHP memory limit, you will change the value of env variable <code class="language-plaintext highlighter-rouge">MEMORY_LIMIT</code>.</p>
<p>Then you will run that kind of command to update your PHP service:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker stack deploy -c docker-compose.yml --with-registry-auth "symfony_application"
</code></pre></div></div>
<p>And <strong>BOOM !!!</strong> All the containers (probably spread over many servers, maybe over many datacenters)
of all your services of your stack will restart following defined <code class="language-plaintext highlighter-rouge">restart_policy</code>, <code class="language-plaintext highlighter-rouge">update_config</code> and <code class="language-plaintext highlighter-rouge">healthcheck</code> configurations.</p>
<p>This will lead to an extra and unnecessary resources consumption (CPU, memory, bandwith, …) and maybe service downtime.
Our example is pretty simple but it’s a common thing to have 5 or 6 services per stack, for instance:</p>
<ul>
<li>Nginx as a web server</li>
<li>PHP-FPM as factCGI process manager</li>
<li>Elasticsearch for full-text search</li>
<li>Redis for caching</li>
<li>MySQL a main database storage</li>
</ul>
<p>We can’t afford to restart everything when we simply want to update a single service.</p>
<h1 id="possibles-solutions">Possibles solutions.</h1>
<p>Choose the one you prefer or the one that best fits your needs.</p>
<h2 id="solution-1-never-use-the-env_file-or---env-file-configuration">Solution #1: Never use the <code class="language-plaintext highlighter-rouge">env_file</code> (or <code class="language-plaintext highlighter-rouge">--env-file</code>) configuration.</h2>
<p>Docker-compose allows us to define environment variables to pass to running containers, with <code class="language-plaintext highlighter-rouge">environment</code> config, this way no other variable will be available in the container:</p>
<script src="https://gist.github.com/ypereirareis/362ddf09620769fb9625d2288249d6be.js"></script>
<h2 id="solution-2-split-your-env-file-into-multiple-env-files">Solution #2: Split your env file into multiple env files.</h2>
<ul>
<li>.env (used by docker-compose)</li>
<li>.php.env (used by php service and application)</li>
<li>.nginx.env (used by nginx service)</li>
<li>…</li>
</ul>
<p>and the matching <code class="language-plaintext highlighter-rouge">docker-compose.yml</code>:</p>
<script src="https://gist.github.com/ypereirareis/4ae63ee240fd4aa65d8ef31b6191514f.js"></script>
<h1 id="another-thing-to-consider">Another thing to consider.</h1>
<p>When building your docker image you may add all your env files in the image if you are not careful.</p>
<p>Just see <code class="language-plaintext highlighter-rouge">.dockerignore</code> file or <code class="language-plaintext highlighter-rouge">RUN rm -f *.env</code></p>
<p><a href="https://ypereirareis.github.io/blog/2019/10/28/why-you-should-split-env-file-with-docker-compose-docker-swarm-stack-and-services/">Why you should split your env file with docker-compose and docker swarm stack and services</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on October 28, 2019.</p>
https://ypereirareis.github.io/blog/2019/07/30/php-fpm-truncated-log-workaround-solution-trick2019-07-30T00:00:00+00:002019-07-30T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p><img src="/images/posts/php-fpm.gif" alt="PHP-FPM" /></p>
<h1 id="php-fpm-truncated-logs-problem">PHP-FPM truncated logs problem</h1>
<p>[BUG] - <a href="https://bugs.php.net/bug.php?id=71880">https://bugs.php.net/bug.php?id=71880</a></p>
<ul>
<li>When you write to STDOUT or STDERR (php://stdout OR php://stderr) PHP-FPM creates a warning in log files.</li>
<li>This is a problem when you want to run PHP-FPM in a docker container. It’s common pratice for docker containers to write any log output to STDOUT/STDERR.</li>
<li>If you do this with e.g. with the official php-fpm docker image, you’ll end up with tons of ugly warnings like above. Right now there’s no way to get rid of these warnings. It will also split up a single multi-line output into several distinct warnings.</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>WARNING: <span class="o">[</span>pool www] child 12 said into stdout: <span class="s2">"my output string..."</span>
</code></pre></div></div>
<h1 id="workaround-trick-or-configuration-for-php-fpm-truncated-logs">Workaround, trick or configuration for PHP-FPM truncated logs</h1>
<h2 id="php-70-71-and-72">PHP 7.0, 7.1 and 7.2</h2>
<p>Nothing can be configured to avoid this problem. If you read the issue referenced in the first part of this article,
you can see that workarounds are possible. Let’s see the one I implemented in my php-fpm containers before php 7.3:</p>
<ul>
<li>Use a named pipe created in the container.</li>
<li>tail the named pipe stream to the container output.</li>
</ul>
<h3 id="docker-entrypoint-script">Docker entrypoint script</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># ...SOME_CODE_HERE...</span>
<span class="c">## Named pipe for application logs.</span>
<span class="c">## Trick for bug : https://bugs.php.net/bug.php?id=71880</span>
<span class="nv">NAMED_PIPED</span><span class="o">=</span><span class="s2">"/tmp/stdout"</span>
<span class="nb">rm</span> <span class="nt">-rf</span> <span class="s2">"</span><span class="nv">$NAMED_PIPED</span><span class="s2">"</span> <span class="o">||</span> <span class="nb">true
mkfifo</span> <span class="nt">--mode</span> 600 <span class="s2">"</span><span class="k">${</span><span class="nv">NAMED_PIPED</span><span class="k">}</span><span class="s2">"</span> <span class="o">||</span> <span class="nb">true
chown</span> <span class="s2">"CORRECT_UID_HERE_OR_MAYBE_www-data"</span> <span class="s2">"</span><span class="k">${</span><span class="nv">NAMED_PIPED</span><span class="k">}</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"Named pipe: (</span><span class="nv">$NAMED_PIPED</span><span class="s2">) creation OK"</span>
<span class="o">(</span><span class="nb">tail</span> <span class="nt">-q</span> <span class="nt">-f</span> <span class="k">${</span><span class="nv">NAMED_PIPED</span><span class="k">}</span> <span class="o">>></span> /proc/self/fd/2 <span class="o">||</span> pkill php-fpm<span class="o">)</span> &
<span class="c"># ...SOME_CODE_HERE...</span>
</code></pre></div></div>
<h3 id="write-to-named-pipe-instead-of-phpstderr-and-phpstdout">Write to named pipe instead of php://stderr and php://stdout</h3>
<p>For instance with monolog you can write that kind of configuration:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">monolog</span><span class="pi">:</span>
<span class="na">handlers</span><span class="pi">:</span>
<span class="na">main</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">stream</span>
<span class="na">path</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/tmp/stdout"</span>
<span class="na">level</span><span class="pi">:</span> <span class="s">error</span>
</code></pre></div></div>
<p><strong>Note that the path in monolog must match the named pipe name.</strong></p>
<h2 id="php-73">PHP 7.3</h2>
<p>You can disable workers output decoration with a simple configuration in <code class="language-plaintext highlighter-rouge">www.conf</code> for the fpm pool.
You will have standard logs without decoration and ugly WARNINGS.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>decorate_workers_output <span class="o">=</span> no
</code></pre></div></div>
<p><a href="https://ypereirareis.github.io/blog/2019/07/30/php-fpm-truncated-log-workaround-solution-trick/">Docker and php-fpm truncated logs workaround and configuration for php 7.0, 7.1, 7.2 and 7.3</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on July 30, 2019.</p>
https://ypereirareis.github.io/blog/2019/06/04/real-client-ip-address-nginx-behind-haproxy-reverse-proxy2019-06-04T00:00:00+00:002019-06-04T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p><img src="/images/posts/haproxy.gif" alt="Docker" /></p>
<h1 id="define-a-header-in-haproxy-configuration-to-set-real-client-ip-address">Define a header in haproxy configuration to set real client IP address</h1>
<p>In haproxy, you can get the client IP address from the request and pass it to another prox or application with a header.</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">%[src]</code> is the client IP address extracted from incoming request.</li>
<li><code class="language-plaintext highlighter-rouge">X-Real-IP</code> is the header we use to transfer IP address value.</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>frontend all_https
option forwardfor header X-Real-IP
http-request set-header X-Real-IP %[src]
</code></pre></div></div>
<h1 id="configure-a-custom-log-format-in-nginx">Configure a custom log format in Nginx</h1>
<ul>
<li>Add a custom log format named “realip” (but name it the way you want…)</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>log_format realip <span class="s1">'$http_x_real_ip - $remote_user [$time_local] '</span>
<span class="s1">'"$request" $status $body_bytes_sent '</span>
<span class="s1">'"$http_referer" "$http_user_agent"'</span><span class="p">;</span>
</code></pre></div></div>
<ul>
<li>Use it in your <code class="language-plaintext highlighter-rouge">access_log</code> directive</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>access_log /dev/stdout realip<span class="p">;</span>
access_log /path_to/log/file realip<span class="p">;</span>
</code></pre></div></div>
<p><a href="https://ypereirareis.github.io/blog/2019/06/04/real-client-ip-address-nginx-behind-haproxy-reverse-proxy/">Get real client IP address in NGINX behind HAPROXY reverse proxy</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on June 04, 2019.</p>
https://ypereirareis.github.io/blog/2019/06/04/haproxy-301-redirect-without-cache2019-06-04T00:00:00+00:002019-06-04T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p><img src="/images/posts/haproxy.gif" alt="Docker" /></p>
<h1 id="example-configuration-for-haproxycfg">Example configuration for haproxy.cfg</h1>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>acl example-1 hdr<span class="o">(</span>host<span class="o">)</span> <span class="nt">-i</span> example-1.com
acl example-2 hdr<span class="o">(</span>host<span class="o">)</span> <span class="nt">-i</span> example-2.com
http-request redirect location https://www.%[hdr<span class="o">(</span>host<span class="o">)]</span>%[url]<span class="se">\r\n</span>Cache-Control:<span class="se">\ </span>no-cache,<span class="se">\ </span>no-store,<span class="se">\ </span>max-age<span class="o">=</span>0,<span class="se">\ </span>must-revalidate code 301 <span class="k">if </span>example-1 or example-2
</code></pre></div></div>
<p><a href="https://ypereirareis.github.io/blog/2019/06/04/haproxy-301-redirect-without-cache/">Doing a (301) redirect with HAPROXY without cache</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on June 04, 2019.</p>
https://ypereirareis.github.io/blog/2018/03/02/mysql-delete-group-by-having2018-03-02T00:00:00+00:002018-03-02T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p><img src="/images/posts/mysql.jpg" alt="Docker" /></p>
<h1 id="mysql-delete-with-group-by-and-having-clauses">MySQL Delete with Group By and Having clauses.</h1>
<p>To achieve this exploit with a single simple MySQL command, we are using two useful functions:</p>
<ul>
<li><strong>1. <a href="https://dev.mysql.com/doc/refman/5.7/en/group-by-functions.html#function_group-concat">GROUP_CONCAT()</a></strong></li>
</ul>
<blockquote>
<p>This function returns a string result with the concatenated non-NULL values from a group. It returns NULL if there are no non-NULL values.</p>
</blockquote>
<ul>
<li><strong>2. <a href="https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_find-in-set">FIND_IN_SET()</a></strong></li>
</ul>
<blockquote>
<p>Returns a value in the range of 1 to N if the string str is in the string list strlist consisting of N substrings. A string list is a string composed of substrings separated by , characters. If the first argument is a constant string and the second is a column of type SET, the FIND_IN_SET() function is optimized to use bit arithmetic. Returns 0 if str is not in strlist or if strlist is the empty string. Returns NULL if either argument is NULL. This function does not work properly if the first argument contains a comma (,) character.</p>
</blockquote>
<h2 id="select-the-ids-we-want-to-delete-with-the-group-by-and-having-clauses">Select the ids we want to delete with the GROUP BY and HAVING clauses</h2>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT group_concat<span class="o">(</span><span class="nb">id</span><span class="o">)</span> AS ids_to_delete
FROM table
GROUP BY field_1, field_2,...
HAVING count<span class="o">(</span><span class="k">*</span><span class="o">)</span> <span class="o">>=</span> 5
</code></pre></div></div>
<p>The result will look like that:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">---------------</span>
ids_to_delete
<span class="nt">---------------</span>
1,34,87,8756,4657,34
</code></pre></div></div>
<h2 id="delete-rows-directly-using-this-list-of-ids">Delete rows directly using this list of ids.</h2>
<p>To do this we use the <code class="language-plaintext highlighter-rouge">FIND_IN_SET()</code> function.</p>
<blockquote>
<p><strong>[EDIT 1] (David Gurba)</strong><br />
<a href="https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_group_concat_max_len">group_concat_max_len</a>:
The maximum permitted result length in bytes for the GROUP_CONCAT() function. The default is 1024.<br />
SET SESSION group_concat_max_len=4294967295;<br />
SET GLOBAL group_concat_max_len=18446744073709551615;</p>
</blockquote>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>DELETE FROM table
WHERE FIND_IN_SET<span class="o">(</span><span class="nb">id</span>, <span class="o">(</span>
SELECT ids_to_delete
FROM <span class="o">(</span>
SELECT group_concat<span class="o">(</span><span class="nb">id</span><span class="o">)</span> AS ids_to_delete
FROM table
GROUP BY field_1, field_2,...
HAVING count<span class="o">(</span><span class="k">*</span><span class="o">)</span> <span class="o">>=</span> 5
<span class="o">)</span> t
<span class="o">))</span>
</code></pre></div></div>
<h2 id="dry-run---if-you-want-to-check-what-you-will-remove-before-really-doing-it">Dry run - If you want to check what you will remove before really doing it:</h2>
<p><code class="language-plaintext highlighter-rouge">DELETE FROM table</code> => <code class="language-plaintext highlighter-rouge">SELECT * FROM table</code></p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT <span class="k">*</span> FROM table
WHERE FIND_IN_SET<span class="o">(</span><span class="nb">id</span>, <span class="o">(</span>
SELECT ids_to_delete
FROM <span class="o">(</span>
SELECT group_concat<span class="o">(</span><span class="nb">id</span><span class="o">)</span> AS ids_to_delete
FROM table
GROUP BY field_1, field_2,...
HAVING count<span class="o">(</span><span class="k">*</span><span class="o">)</span> <span class="o">>=</span> 5
<span class="o">)</span> t
<span class="o">))</span>
</code></pre></div></div>
<p><strong>That’s all ;)</strong></p>
<p><a href="https://ypereirareis.github.io/blog/2018/03/02/mysql-delete-group-by-having/">How to delete data in MYSQL with DELETE, GROUP BY and HAVING clauses</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on March 02, 2018.</p>
https://ypereirareis.github.io/blog/2017/04/25/vuejs-two-way-data-binding-state-management-vuex-strict-mode2017-04-25T00:00:00+00:002017-04-25T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p><img src="/images/posts/vuejs.jpg" alt="Docker" /></p>
<p><strong>In the following code samples we are using ES6/ES2015/Babel.</strong></p>
<h1 id="two-way-data-binding-with-vuejs-and-vuex">Two way data binding with VueJS and Vuex</h1>
<p>If you don’t really know what two way data binding is, let’s see a few definitions:</p>
<blockquote>
<p>Two way binding means that any data-related changes affecting the model are immediately propagated to the matching view(s),
and that any changes made in the view(s) (say, by the user) are immediately reflected in the underlying model.
When app data changes, so does the UI, and conversely.</p>
</blockquote>
<blockquote>
<p>Two way binding just means that,
when properties in the model get updated, so does the UI.
When UI elements get updated, the changes get propagated back to the model.</p>
</blockquote>
<p>It’s possible to use two way binding with javascript frameworks like AngularJs, ReactJs, VueJs….</p>
<h2 id="setup-statestore-with-vuex">Setup state/store with Vuex</h2>
<p>Of course, to setup Vuex with Vuejs you need two things:</p>
<ul>
<li>A fully working VueJs project.</li>
<li>Following <a href="https://vuex.vuejs.org/en/">Vuex documentation</a>, beginning with the <a href="https://vuex.vuejs.org/en/installation.html">installation part</a>.</li>
</ul>
<h3 id="lets-see-a-very-simple-vuejsvuex-configuration">Let’s see a very simple VueJS/Vuex configuration</h3>
<p>Click on the <em>Result</em> tab to see “VueJs two way binding” in action.</p>
<p><strong>The external resources are important:</strong></p>
<ul>
<li><a href="https://cdnjs.cloudflare.com/ajax/libs/vue/2.2.6/vue.min.js">https://cdnjs.cloudflare.com/ajax/libs/vue/2.2.6/vue.min.js</a></li>
<li><a href="https://cdnjs.cloudflare.com/ajax/libs/vuex/2.3.1/vuex.min.js">https://cdnjs.cloudflare.com/ajax/libs/vuex/2.3.1/vuex.min.js</a></li>
</ul>
<script async="" src="//jsfiddle.net/ypereirareis/cpg40rh3/embed/js,html,result/dark/"></script>
<p>In this first example, if you open the debug tool of your browser (console), you should see this error message.</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">vue</span><span class="p">.</span><span class="nx">min</span><span class="p">.</span><span class="nx">js</span><span class="p">:</span><span class="mi">6</span> <span class="nb">Error</span><span class="p">:</span> <span class="p">[</span><span class="nx">vuex</span><span class="p">]</span> <span class="nx">Do</span> <span class="nx">not</span> <span class="nx">mutate</span> <span class="nx">vuex</span> <span class="nx">store</span> <span class="nx">state</span> <span class="nx">outside</span> <span class="nx">mutation</span> <span class="nx">handlers</span><span class="p">.</span>
<span class="nx">at</span> <span class="nx">r</span> <span class="p">(</span><span class="nx">vuex</span><span class="p">.</span><span class="nx">min</span><span class="p">.</span><span class="nx">js</span><span class="p">:</span><span class="mi">6</span><span class="p">)</span>
<span class="nx">at</span> <span class="nx">nt</span><span class="p">.</span><span class="nx">t</span><span class="p">.</span><span class="nx">_vm</span><span class="p">.</span><span class="nx">$watch</span><span class="p">.</span><span class="nx">deep</span> <span class="p">(</span><span class="nx">vuex</span><span class="p">.</span><span class="nx">min</span><span class="p">.</span><span class="nx">js</span><span class="p">:</span><span class="mi">6</span><span class="p">)</span>
<span class="nx">at</span> <span class="nx">ho</span><span class="p">.</span><span class="nx">run</span> <span class="p">(</span><span class="nx">vue</span><span class="p">.</span><span class="nx">min</span><span class="p">.</span><span class="nx">js</span><span class="p">:</span><span class="mi">7</span><span class="p">)</span>
<span class="nx">at</span> <span class="nx">ho</span><span class="p">.</span><span class="nx">update</span> <span class="p">(</span><span class="nx">vue</span><span class="p">.</span><span class="nx">min</span><span class="p">.</span><span class="nx">js</span><span class="p">:</span><span class="mi">7</span><span class="p">)</span>
<span class="nx">at</span> <span class="nx">qi</span><span class="p">.</span><span class="nx">notify</span> <span class="p">(</span><span class="nx">vue</span><span class="p">.</span><span class="nx">min</span><span class="p">.</span><span class="nx">js</span><span class="p">:</span><span class="mi">7</span><span class="p">)</span>
<span class="nx">at</span> <span class="nb">Object</span><span class="p">.</span><span class="kd">set</span> <span class="p">[</span><span class="k">as</span> <span class="nx">lastname</span><span class="p">]</span> <span class="p">(</span><span class="nx">vue</span><span class="p">.</span><span class="nx">min</span><span class="p">.</span><span class="nx">js</span><span class="p">:</span><span class="mi">6</span><span class="p">)</span>
<span class="nx">at</span> <span class="nx">input</span> <span class="p">(</span><span class="nb">eval</span> <span class="nx">at</span> <span class="nx">li</span> <span class="p">(</span><span class="nx">vue</span><span class="p">.</span><span class="nx">min</span><span class="p">.</span><span class="nx">js</span><span class="p">:</span><span class="mi">1</span><span class="p">),</span> <span class="o"><</span><span class="nx">anonymous</span><span class="o">></span><span class="p">:</span><span class="mi">2</span><span class="p">:</span><span class="mi">416</span><span class="p">)</span>
<span class="nx">at</span> <span class="nx">HTMLInputElement</span><span class="p">.</span><span class="nx">t</span> <span class="p">(</span><span class="nx">vue</span><span class="p">.</span><span class="nx">min</span><span class="p">.</span><span class="nx">js</span><span class="p">:</span><span class="mi">6</span><span class="p">)</span></code></pre></figure>
<p>This error is throw by Vuex, because we have enabled the <a href="https://vuex.vuejs.org/en/strict.html">Vuex strict mode</a>.</p>
<blockquote>
<p>This ensures that all state mutations can be explicitly tracked by debugging tools.</p>
</blockquote>
<p>We’ll talk about this later in the article…</p>
<h2 id="is-two-way-data-binding-a-best-practice-or-not-">Is two way data binding a best practice or not ?</h2>
<h3 id="i-see-one-big-advantage-to-use-two-way-binding">I see one big advantage to use two way binding:</h3>
<ul>
<li>Real time view updates thanks to Virtual DOM (very fast and not (re)rendering sub components if not needed).</li>
</ul>
<p>It’s perfectly suitable for small applications with not too much real time state updates.</p>
<h3 id="but-i-see-many-drawbacks-too">But I see many drawbacks too:</h3>
<ul>
<li>Watchers needed to update view when model is updated.</li>
<li>No way to track model updates in a centralized place.</li>
<li>Many places where state can be updated.</li>
<li>No way (not really true because we can rely on component watchers) to debounce or filter (or whatever) updates on state.</li>
</ul>
<h2 id="vuex-strict-mode">Vuex strict mode</h2>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">const</span> <span class="nx">store</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Vuex</span><span class="p">.</span><span class="nx">Store</span><span class="p">({</span>
<span class="c1">// ...</span>
<span class="na">strict</span><span class="p">:</span> <span class="kc">true</span>
<span class="p">})</span></code></pre></figure>
<ul>
<li>This ensures that all state mutations can be explicitly tracked by debugging tools.</li>
<li>Whenever Vuex state is mutated outside of mutation handlers, an error will be thrown.</li>
<li>Strict mode runs a synchronous deep watcher on the state tree for detecting inappropriate mutations, and it can be quite expensive when you make large amount of mutations to the state. Make sure to turn it off in production to avoid the performance cost.</li>
</ul>
<h1 id="solutions-to-track-state-changes-and-improve-performance-with-vuex">Solutions to track state changes and improve performance with Vuex</h1>
<p>To remove the Vuex error and update the state in a mutation without adding or updating a lot of code,
an option can be to deep clone the object before updating it with two way binding in our form.</p>
<h2 id="deep-clone-and-watch">Deep clone and watch</h2>
<p>The following few lines show how to deep clone the object and how to see updates on the object and on the deep copy.
We are using <a href="https://lodash.com/docs/4.17.4#cloneDeep">lodash</a> to clone the object.</p>
<script async="" src="//jsfiddle.net/ypereirareis/p1rwn9rb/embed/js,html,result/dark/"></script>
<h3 id="what-are-we-doing-here-">What are we doing here ?</h3>
<ul>
<li>We simply bypass the Vuex error.</li>
<li>We are still using two way binding, but with the cloned object and not the original one.</li>
<li>We must add a watcher/handler on this cloned object to track updates.</li>
<li>We add a mutation to update the state at the end of the process.</li>
<li>In the watcher/handler we can call the mutation to update the source object.</li>
</ul>
<p><strong>The computed property and the debounced mutation call, allows to track the effective state updates</strong></p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">computed</span><span class="p">:</span> <span class="p">{</span>
<span class="nx">userState</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">$store</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">user</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nl">handler</span><span class="p">:</span> <span class="nx">_</span><span class="p">.</span><span class="nx">debounce</span><span class="p">(</span><span class="kd">function</span> <span class="p">(</span><span class="nx">user</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">$store</span><span class="p">.</span><span class="nx">commit</span><span class="p">(</span><span class="dl">'</span><span class="s1">updateUser</span><span class="dl">'</span><span class="p">,</span> <span class="nx">user</span><span class="p">);</span>
<span class="p">},</span> <span class="mi">500</span><span class="p">),</span> <span class="nx">deep</span><span class="p">:</span> <span class="kc">true</span></code></pre></figure>
<p><strong>Be careful !</strong> You cannot debounce the function in the mutation itself.
Indeed, the code would be executed within the next event loop cycle, not really in the mutation function.
The Vuex error will appear again.</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"> <span class="nx">mutations</span><span class="p">:</span> <span class="p">{</span>
<span class="nl">updateUser</span><span class="p">:</span> <span class="nx">_</span><span class="p">.</span><span class="nx">debounce</span><span class="p">(</span><span class="kd">function</span> <span class="p">(</span><span class="nx">state</span><span class="p">,</span> <span class="nx">user</span><span class="p">)</span> <span class="p">{</span>
<span class="nb">Object</span><span class="p">.</span><span class="nx">assign</span><span class="p">(</span><span class="nx">state</span><span class="p">.</span><span class="nx">user</span><span class="p">,</span> <span class="nx">user</span><span class="p">);</span>
<span class="p">},</span><span class="mi">500</span><span class="p">)</span>
<span class="p">}</span></code></pre></figure>
<p>The “deep clone” solution is not really a perfect solution because we are still using two way binding,
and we are adding a watcher manually which has performance cost. <strong>So… how can we improve the situation ?</strong></p>
<h2 id="one-way-data-binding-and-explicit-data-update">One way data binding and explicit data update</h2>
<p>Actually, two way binding is not something really needed in most cases.
What about removing it and relying on one way binding and explicit data updates ?</p>
<script async="" src="//jsfiddle.net/ypereirareis/x2gs3ha4/embed/js,html,result/dark/"></script>
<h3 id="what-are-we-doing-here--1">What are we doing here ?</h3>
<ul>
<li>We are no more cloning the object.</li>
<li>We are using one way binding and explicitly updating object properties.</li>
</ul>
<figure class="highlight"><pre><code class="language-html" data-lang="html"><span class="nt"><input</span> <span class="na">:value=</span><span class="s">"user.lastname"</span> <span class="na">v-on:keyup.stop=</span><span class="s">"updateLastname($event.target.value)"</span> <span class="nt">/></span>
<span class="nt"><input</span> <span class="na">:value=</span><span class="s">"user.firstname"</span> <span class="na">v-on:keyup.stop=</span><span class="s">"updateFirstname($event.target.value)"</span><span class="nt">/></span></code></pre></figure>
<ul>
<li>We need methods to update object properties.</li>
</ul>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">methods</span><span class="p">:</span> <span class="p">{</span>
<span class="nx">updateFirstname</span><span class="p">(</span><span class="nx">firstname</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">$store</span><span class="p">.</span><span class="nx">commit</span><span class="p">(</span><span class="dl">'</span><span class="s1">updateUser</span><span class="dl">'</span><span class="p">,</span> <span class="p">{</span><span class="nx">firstname</span><span class="p">});</span>
<span class="p">},</span>
<span class="nx">updateLastname</span><span class="p">(</span><span class="nx">lastname</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">$store</span><span class="p">.</span><span class="nx">commit</span><span class="p">(</span><span class="dl">'</span><span class="s1">updateUser</span><span class="dl">'</span><span class="p">,</span> <span class="p">{</span><span class="nx">lastname</span><span class="p">});</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p><strong>And… that’s it !</strong></p>
<h3 id="after-a-small-refactoring">After a small refactoring</h3>
<p>It’s possible to refactor the code to have a single method to update object properties.</p>
<script async="" src="//jsfiddle.net/ypereirareis/cve0dtkb/embed/js,html,result/dark/"></script>
<p>The interesting parts here are:</p>
<figure class="highlight"><pre><code class="language-html" data-lang="html"><span class="nt"><input</span> <span class="na">:value=</span><span class="s">"user.lastname"</span> <span class="na">v-on:keyup.stop=</span><span class="s">"updateField('lastname', $event.target.value)"</span> <span class="nt">/></span>
<span class="nt"><input</span> <span class="na">:value=</span><span class="s">"user.firstname"</span> <span class="na">v-on:keyup.stop=</span><span class="s">"updateField('firstname', $event.target.value)"</span><span class="nt">/></span></code></pre></figure>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">methods</span><span class="p">:</span> <span class="p">{</span>
<span class="nx">updateField</span><span class="p">(</span><span class="nx">field</span><span class="p">,</span> <span class="nx">value</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">$store</span><span class="p">.</span><span class="nx">commit</span><span class="p">(</span><span class="dl">'</span><span class="s1">updateUser</span><span class="dl">'</span><span class="p">,</span> <span class="p">{</span>
<span class="p">[</span><span class="nx">field</span><span class="p">]:</span> <span class="nx">value</span>
<span class="p">});</span>
<span class="p">},</span>
<span class="p">}</span></code></pre></figure>
<h3 id="refactor-again-and-again">Refactor again and again</h3>
<p>If we are updating a user in another VueJs component we need to duplicate this part of the code:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">methods</span><span class="p">:</span> <span class="p">{</span>
<span class="nx">updateField</span><span class="p">(</span><span class="nx">field</span><span class="p">,</span> <span class="nx">value</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">$store</span><span class="p">.</span><span class="nx">commit</span><span class="p">(</span><span class="dl">'</span><span class="s1">updateUser</span><span class="dl">'</span><span class="p">,</span> <span class="p">{</span>
<span class="p">[</span><span class="nx">field</span><span class="p">]:</span> <span class="nx">value</span>
<span class="p">});</span>
<span class="p">},</span>
<span class="p">}</span></code></pre></figure>
<p>What about moving the computed property key of the object literal in the mutation itself ?</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">methods</span><span class="p">:</span> <span class="p">{</span>
<span class="nx">updateField</span><span class="p">(</span><span class="nx">field</span><span class="p">,</span> <span class="nx">value</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">$store</span><span class="p">.</span><span class="nx">commit</span><span class="p">(</span><span class="dl">'</span><span class="s1">updateUser</span><span class="dl">'</span><span class="p">,</span> <span class="p">{</span><span class="nx">field</span><span class="p">,</span> <span class="nx">value</span><span class="p">});</span>
<span class="p">},</span>
<span class="p">}</span>
<span class="nl">mutations</span><span class="p">:</span> <span class="p">{</span>
<span class="nl">updateUser</span><span class="p">:</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">state</span><span class="p">,</span> <span class="p">{</span><span class="nx">field</span><span class="p">,</span> <span class="nx">value</span><span class="p">})</span> <span class="p">{</span>
<span class="nb">Object</span><span class="p">.</span><span class="nx">assign</span><span class="p">(</span><span class="nx">state</span><span class="p">.</span><span class="nx">user</span><span class="p">,</span> <span class="p">{</span>
<span class="p">[</span><span class="nx">field</span><span class="p">]:</span> <span class="nx">value</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<h2 id="edit-two-way-computed-property">[EDIT] Two-way Computed Property</h2>
<p>You could use a built-in VueJs functionality describred at the end of <a href="https://vuex.vuejs.org/en/forms.html">this page</a>.
But once again <strong>be careful</strong> because it works only for simple computed properties, not for object (with many depth levels potentially).</p>
<p>So if you have an object with 10 properties to update, you will need to set 10 two way computed properties (each with a get and a set method). It’s gonna be a lot more verbose than the previous solution.</p>
<p><strong>This kinda configuration</strong></p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">computed</span><span class="p">:</span> <span class="p">{</span>
<span class="nl">prop1</span><span class="p">:</span> <span class="p">{</span>
<span class="kd">get</span> <span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">$store</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">obj</span><span class="p">.</span><span class="nx">prop1</span>
<span class="p">},</span>
<span class="kd">set</span> <span class="p">(</span><span class="nx">value</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">$store</span><span class="p">.</span><span class="nx">commit</span><span class="p">(</span><span class="dl">'</span><span class="s1">updateprop1</span><span class="dl">'</span><span class="p">,</span> <span class="nx">value</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="nx">prop2</span><span class="p">:</span> <span class="p">{</span>
<span class="kd">get</span> <span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">$store</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">obj</span><span class="p">.</span><span class="nx">prop2</span>
<span class="p">},</span>
<span class="kd">set</span> <span class="p">(</span><span class="nx">value</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">$store</span><span class="p">.</span><span class="nx">commit</span><span class="p">(</span><span class="dl">'</span><span class="s1">updateprop2</span><span class="dl">'</span><span class="p">,</span> <span class="nx">value</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="p">...</span>
<span class="p">,</span>
<span class="nx">prop10</span><span class="p">:</span> <span class="p">{</span>
<span class="kd">get</span> <span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">$store</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">obj</span><span class="p">.</span><span class="nx">prop10</span>
<span class="p">},</span>
<span class="kd">set</span> <span class="p">(</span><span class="nx">value</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">$store</span><span class="p">.</span><span class="nx">commit</span><span class="p">(</span><span class="dl">'</span><span class="s1">updateprop10</span><span class="dl">'</span><span class="p">,</span> <span class="nx">value</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<h1 id="conclusion">Conclusion</h1>
<p>Two way data binding is really easy to setup with all major javascript frameworks (it’s often the default behavior).
It’s a very good option for small applications or POC.
But for very complex UI you should consider using one way data binding and explicit state updates/mutations.</p>
<p><a href="https://ypereirareis.github.io/blog/2017/04/25/vuejs-two-way-data-binding-state-management-vuex-strict-mode/">VueJS - Two way data binding and state management with Vuex and strict mode</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on April 25, 2017.</p>
https://ypereirareis.github.io/blog/2017/04/03/rabbitmq-high-available-cluster-haproxy-docker2017-04-03T00:00:00+00:002017-04-03T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p><img src="/images/posts/rabbitmq.jpg" alt="Docker" /></p>
<h1 id="tldr">TL;DR</h1>
<p>Docker based project to run a highly available RabbitMQ cluster:
<a href="https://github.com/ypereirareis/docker-rabbitmq-ha-cluster">https://github.com/ypereirareis/docker-rabbitmq-ha-cluster</a></p>
<h1 id="rabbitmq-cluster">RabbitMQ cluster</h1>
<p>A cluster is composed of at least two RabbitMQ nodes. These nodes need to be able to communicate with each other.
<strong>I strongly advise you to always have an odd number of nodes in your cluster.</strong></p>
<h1 id="load-balancing-with-haproxy">Load Balancing with HAProxy</h1>
<p><img src="/images/posts/haproxy.gif" alt="Docker" /></p>
<p>When working with a cluster the goal is to have a highly available service.
So we need to dispatch requests on every running node of the cluster, and avoid sending request to failing nodes.</p>
<p>One way to achieve this is to use HAProxy. It’s a very light and very good tool when dealing with reverse proxy or load balancing.
HAProxy allows TCP connections and redirections out of the box and works well with the AMQP protocol.
<strong>With NGINX you will need to install plugins to manage AMQP connections.</strong></p>
<p>The HAProxy service <strong>SHOULD NOT</strong> be run on a node of the RAbbitMQ cluster. Because if the node fails, the load balancer will fail too.
And you’ll loose the ability to load balance requests on other nodes.</p>
<p><strong>Configuration example</strong></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
log-send-hostname
maxconn 4096
pidfile /var/run/haproxy.pid
user haproxy
group haproxy
daemon
stats socket /var/run/haproxy.stats level admin
ssl-default-bind-options no-sslv3
ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:DHE-DSS-AES128-SHA:DES-CBC3-SHA
defaults
balance roundrobin
log global
mode tcp
option redispatch
option httplog
option dontlognull
option forwardfor
<span class="nb">timeout </span>connect 5000
<span class="nb">timeout </span>client 50000
<span class="nb">timeout </span>server 50000
listen stats
<span class="nb">bind</span> :1936
mode http
stats <span class="nb">enable
timeout </span>connect 10s
<span class="nb">timeout </span>client 1m
<span class="nb">timeout </span>server 1m
stats hide-version
stats realm Haproxy<span class="se">\ </span>Statistics
stats uri /
stats auth stats:stats
listen port_5672
<span class="nb">bind</span> :5672
mode tcp
server rmq_rmq3_1 rmq_rmq3_1:5672 check inter 2000 rise 2 fall 3
server rmq_rmq2_1 rmq_rmq2_1:5672 check inter 2000 rise 2 fall 3
server rmq_rmq1_1 rmq_rmq1_1:5672 check inter 2000 rise 2 fall 3
listen port_15672
<span class="nb">bind</span> :15672
mode tcp
server rmq_rmq1_1 rmq_rmq1_1:15672 check inter 2000 rise 2 fall 3
frontend default_port_80
<span class="nb">bind</span> :80
reqadd X-Forwarded-Proto:<span class="se">\ </span>http
maxconn 4096
default_backend default_service
backend default_service
server rmq_rmq1_1 rmq_rmq1_1:25672 check inter 2000 rise 2 fall 3
server rmq_rmq1_1 rmq_rmq1_1:4369 check inter 2000 rise 2 fall 3
server rmq_rmq1_1 rmq_rmq1_1:9100 check inter 2000 rise 2 fall 3
server rmq_rmq1_1 rmq_rmq1_1:9101 check inter 2000 rise 2 fall 3
server rmq_rmq1_1 rmq_rmq1_1:9102 check inter 2000 rise 2 fall 3
server rmq_rmq1_1 rmq_rmq1_1:9103 check inter 2000 rise 2 fall 3
server rmq_rmq1_1 rmq_rmq1_1:9104 check inter 2000 rise 2 fall 3
server rmq_rmq1_1 rmq_rmq1_1:9105 check inter 2000 rise 2 fall 3
server rmq_rmq2_1 rmq_rmq2_1:15672 check inter 2000 rise 2 fall 3
server rmq_rmq2_1 rmq_rmq2_1:25672 check inter 2000 rise 2 fall 3
server rmq_rmq2_1 rmq_rmq2_1:4369 check inter 2000 rise 2 fall 3
server rmq_rmq2_1 rmq_rmq2_1:9100 check inter 2000 rise 2 fall 3
server rmq_rmq2_1 rmq_rmq2_1:9101 check inter 2000 rise 2 fall 3
server rmq_rmq2_1 rmq_rmq2_1:9102 check inter 2000 rise 2 fall 3
server rmq_rmq2_1 rmq_rmq2_1:9103 check inter 2000 rise 2 fall 3
server rmq_rmq2_1 rmq_rmq2_1:9104 check inter 2000 rise 2 fall 3
server rmq_rmq2_1 rmq_rmq2_1:9105 check inter 2000 rise 2 fall 3
server rmq_rmq3_1 rmq_rmq3_1:15672 check inter 2000 rise 2 fall 3
server rmq_rmq3_1 rmq_rmq3_1:25672 check inter 2000 rise 2 fall 3
server rmq_rmq3_1 rmq_rmq3_1:4369 check inter 2000 rise 2 fall 3
server rmq_rmq3_1 rmq_rmq3_1:9100 check inter 2000 rise 2 fall 3
server rmq_rmq3_1 rmq_rmq3_1:9101 check inter 2000 rise 2 fall 3
server rmq_rmq3_1 rmq_rmq3_1:9102 check inter 2000 rise 2 fall 3
server rmq_rmq3_1 rmq_rmq3_1:9103 check inter 2000 rise 2 fall 3
server rmq_rmq3_1 rmq_rmq3_1:9104 check inter 2000 rise 2 fall 3
server rmq_rmq3_1 rmq_rmq3_1:9105 check inter 2000 rise 2 fall 3</code></pre></figure>
<h1 id="high-availability">High availability</h1>
<h2 id="nodes">Nodes</h2>
<p>To have high availability you need more than one node, and of course you need load balancing.
Let’s say that 3 nodes is a good start for a simple RAbbitMQ cluster.
And we need one more node for HAProxy.</p>
<p><img src="https://github.com/ypereirareis/docker-rabbitmq-ha-cluster/raw/master/img/rabbitmq.png" alt="Docker" /></p>
<h2 id="exchanges-queues-and-messages-mirroring-and-persistence">Exchanges, Queues and messages mirroring and persistence</h2>
<p>If we want a high level of resiliency, something important is to mirror everything we can on other nodes of the cluster.</p>
<p>We need to configure a few things:</p>
<ul>
<li>durable, mirrored and persistent Exchanges</li>
<li>durable, mirrored and persistent Queues</li>
<li>persistent and mirrored messages (read/write on disk)</li>
</ul>
<p>Now we can have consumers and producers connected with one (or more) nodes of the cluster.</p>
<h1 id="network-partition">Network partition</h1>
<p>Sometimes a node can be excluded and unreachable by the others (network failure for instance).
But it’s still running and receiving / consuming messages.
After a small period of time the node becomes desynchronized, and it appears a network partition.
Messages in the excluded node are not in the others and vice versa.</p>
<h1 id="test-and-benchmark">Test and benchmark</h1>
<p>If you want to experiment scenarios with RabbitMQ cluster, I created a docker based project for that.</p>
<p><a href="https://github.com/ypereirareis/docker-rabbitmq-ha-cluster">https://github.com/ypereirareis/docker-rabbitmq-ha-cluster</a></p>
<p>You can experiment:</p>
<ul>
<li>Load Balancing</li>
<li>Node failure</li>
<li>Network partition</li>
<li>Messages persistency</li>
<li>Message NO ACK and retries</li>
<li>Exchanges and queues durability and mirroring</li>
<li>Polling VS pulling</li>
<li>Swarrot / SwarrotBundle</li>
<li>RabbitMqBundle</li>
</ul>
<p><a href="https://ypereirareis.github.io/blog/2017/04/03/rabbitmq-high-available-cluster-haproxy-docker/">RabbitMQ high available cluster with docker and HAProxy</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on April 03, 2017.</p>
https://ypereirareis.github.io/blog/2017/02/20/php-fpm-cli-error-log2017-02-20T00:00:00+00:002017-02-20T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p><img src="/images/posts/php-fpm.gif" alt="Docker" /></p>
<h1 id="php-fpm">PHP FPM</h1>
<p><strong>First of all we need to enable option <code class="language-plaintext highlighter-rouge">catch_workers_output</code> for fpm.</strong></p>
<blockquote>
<p>catch_workers_output <strong>boolean</strong> <br />
Redirect worker stdout and stderr into main error log. If not set, stdout and stderr will be redirected to /dev/null according to FastCGI specs. Default value: no.</p>
</blockquote>
<p><em>/usr/local/etc/php-fpm.d/www.conf (in my configuration)</em></p>
<ul>
<li><code class="language-plaintext highlighter-rouge">catch_workers_output = yes</code></li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sed</span> <span class="nt">-i</span> <span class="s1">'/^;catch_workers_output/ccatch_workers_output = yes'</span> <span class="s2">"/usr/local/etc/php-fpm.d/www.conf"</span></code></pre></figure>
<p>Or simply edit and save the file manually to uncomment line starting with <code class="language-plaintext highlighter-rouge">;catch_workers_output</code>.</p>
<p><strong>Then we need to configure log file names and locations.</strong></p>
<h2 id="access-log">Access log</h2>
<p>If you want or need to activate access log at php level:</p>
<blockquote>
<p>access.log <strong>string</strong><br />
The access log file. Default value: not set</p>
</blockquote>
<ul>
<li><code class="language-plaintext highlighter-rouge">access.log = /var/log/php/fpm-access.log</code></li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sed</span> <span class="nt">-i</span> <span class="s1">'/^;access.log/caccess.log = /var/log/php/fpm-access.log'</span> <span class="s2">"/usr/local/etc/php-fpm.d/www.conf"</span></code></pre></figure>
<p>Or simply edit and save the file manually to uncomment line starting with <code class="language-plaintext highlighter-rouge">;access.log</code>.</p>
<p>You will have this kind of output:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>tailf var/logs/php/fpm-access.log
172.18.0.5 - 20/Feb/2017:13:07:39 +0100 <span class="s2">"GET /app_dev.php"</span> 200
172.18.0.5 - 20/Feb/2017:13:07:47 +0100 <span class="s2">"POST /app_dev.php"</span> 302
172.18.0.5 - 20/Feb/2017:13:07:47 +0100 <span class="s2">"POST /app_dev.php"</span> 302
172.18.0.5 - 20/Feb/2017:13:07:47 +0100 <span class="s2">"GET /app_dev.php"</span> 200
172.18.0.5 - 20/Feb/2017:13:07:48 +0100 <span class="s2">"GET /app_dev.php"</span> 302
172.18.0.5 - 20/Feb/2017:13:07:48 +0100 <span class="s2">"GET /app_dev.php"</span> 200</code></pre></figure>
<h2 id="error-log">Error log</h2>
<p>Of course in production we do not want to display errors to users:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">php_flag[display_errors] = off</code></li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sed</span> <span class="nt">-i</span> <span class="s1">'/^;php_flag\[display_errors\]/cphp_flag[display_errors] = off'</span> <span class="s2">"/usr/local/etc/php-fpm.d/www.conf"</span></code></pre></figure>
<p>Or simply edit and save the file manually to uncomment line starting with <code class="language-plaintext highlighter-rouge">;php_flag[display_errors]</code>.</p>
<p>Then we must enable error log and define the error log file location :</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">php_admin_value[error_log] = /var/log/php/fpm-error.log</code></li>
<li><code class="language-plaintext highlighter-rouge">php_admin_flag[log_errors] = on</code></li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sed</span> <span class="nt">-i</span> <span class="s1">'/^;php_admin_value\[error_log\]/cphp_admin_value[error_log] = /var/log/php/fpm-error.log'</span> <span class="s2">"/usr/local/etc/php-fpm.d/www.conf"</span>
<span class="nb">sed</span> <span class="nt">-i</span> <span class="s1">'/^;php_admin_flag\[log_errors\]/cphp_admin_flag[log_errors] = on'</span> <span class="s2">"/usr/local/etc/php-fpm.d/www.conf"</span></code></pre></figure>
<p>Or simply edit and save the file manually to uncomment lines starting with <code class="language-plaintext highlighter-rouge">;php_admin_value[error_log]</code> and <code class="language-plaintext highlighter-rouge">;php_admin_flag[log_errors]</code>.</p>
<p>You will have this kind of output:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>tailf var/logs/php/fpm-error.log
<span class="o">[</span>20-Feb-2017 13:33:46 Europe/Paris] PHP Parse error: syntax error, unexpected <span class="s1">'8'</span> <span class="o">(</span>T_LNUMBER<span class="o">)</span>, expecting variable <span class="o">(</span>T_VARIABLE<span class="o">)</span> or <span class="s1">'{'</span> or <span class="s1">'$'</span> <span class="k">in</span> /var/www/html/web/app_dev.php on line 26</code></pre></figure>
<p>You also could change log level:</p>
<blockquote>
<p>log_level <strong>string</strong><br />
Error log level. Possible values: alert, error, warning, notice, debug. Default value: notice.</p>
</blockquote>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sed</span> <span class="nt">-i</span> <span class="s1">'/^;log_level/clog_level = error'</span> <span class="s2">"/usr/local/etc/php-fpm.d/www.conf"</span></code></pre></figure>
<h2 id="important">Important</h2>
<p>Log files must have correct access rights (owner) and must exist:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">mkdir</span> <span class="nt">-p</span> /var/log/php
<span class="nb">touch</span> /var/log/php/fpm-access.log
<span class="nb">touch</span> /var/log/php/fpm-error.log
<span class="nb">chown</span> <span class="nt">-R</span> www-data:www-data /var/log/php</code></pre></figure>
<h1 id="php-cli">PHP CLI</h1>
<p>To enable php CLI errors, we need to add these lines into the (cli) php.ini file.</p>
<p><em>This configuration is for production not for debug or development.</em></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">error_reporting <span class="o">=</span> E_ALL
display_startup_errors <span class="o">=</span> Off
ignore_repeated_errors <span class="o">=</span> Off
ignore_repeated_source <span class="o">=</span> Off
html_errors <span class="o">=</span> Off
track_errors <span class="o">=</span> Off
display_errors <span class="o">=</span> Off
log_errors <span class="o">=</span> On
error_log <span class="o">=</span> /var/log/php/cli-error.log</code></pre></figure>
<h1 id="conclusion">Conclusion</h1>
<p>Using the given configuration you should have those logs:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>ll var/logs/php
total 256K
<span class="nt">-rw-r--r--</span> 1 82 82 0 févr. 17 09:35 cli-error.log
<span class="nt">-rw-r--r--</span> 1 82 82 64K févr. 20 13:34 fpm-access.log
<span class="nt">-rw-r--r--</span> 1 82 82 186 févr. 20 13:33 fpm-error.log</code></pre></figure>
<p>Do NOT forget to enable log rotation, you will have:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>ll var/logs/php
total 256K
<span class="nt">-rw-r--r--</span> 1 82 82 0 févr. 17 09:35 cli-error.log
<span class="nt">-rw-r--r--</span> 1 82 82 390 févr. 17 09:35 cli-error.log-20170217
<span class="nt">-rw-r--r--</span> 1 82 82 64K févr. 20 13:34 fpm-access.log
<span class="nt">-rw-r--r--</span> 1 82 82 100 févr. 17 09:35 fpm-access.log-20170217.gz
<span class="nt">-rw-r--r--</span> 1 82 82 172K févr. 18 02:00 fpm-access.log-20170218
<span class="nt">-rw-r--r--</span> 1 82 82 186 févr. 20 13:33 fpm-error.log
<span class="nt">-rw-r--r--</span> 1 82 82 374 févr. 17 09:35 fpm-error.log-20170217</code></pre></figure>
<p><a href="https://ypereirareis.github.io/blog/2017/02/20/php-fpm-cli-error-log/">PHP fpm and cli error log configuration</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on February 20, 2017.</p>
https://ypereirareis.github.io/blog/2017/02/15/nginx-real-ip-behind-nginx-reverse-proxy2017-02-15T00:00:00+00:002017-02-15T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p>Behind a reverse proxy, the user IP we get is often the reverse proxy IP itself. But for obvious reasons it’s important to have access to the user real ip address.</p>
<p><img src="/images/posts/nginx.gif" alt="Docker" /></p>
<h1 id="nging-reverse-proxy-configuration">Nging reverse proxy configuration</h1>
<p><em>Tested for nginx/1.11.8</em></p>
<p>The <code class="language-plaintext highlighter-rouge">http_realip_module</code> must be installed (<code class="language-plaintext highlighter-rouge">--with-http_realip_module</code>), of course !</p>
<p>Use this command to check :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2>&1 nginx -V | tr -- - '\n' | grep http_realip_module
</code></pre></div></div>
<ul>
<li>We need to tell the reverse proxy to pass information to the backend nginx server.</li>
<li>We can add thoses lines as a global configuration or per location.</li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">proxy_set_header X-Real-IP <span class="nv">$remote_addr</span><span class="p">;</span>
proxy_set_header X-Forwarded-For <span class="nv">$proxy_add_x_forwarded_for</span><span class="p">;</span>
proxy_set_header X-Forwarded-Proto <span class="nv">$proxy_x_forwarded_proto</span><span class="p">;</span></code></pre></figure>
<h1 id="nginx-backend-configuration">Nginx backend configuration</h1>
<ul>
<li>We can add a custom log format and use it in addition with others.</li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">http <span class="o">{</span>
<span class="c"># ...</span>
<span class="c">##</span>
<span class="c"># Logging Settings</span>
<span class="c">##</span>
log_format specialLog <span class="s1">'$remote_addr forwarded for $http_x_real_ip - $remote_user [$time_local] '</span>
<span class="s1">'"$request" $status $body_bytes_sent '</span>
<span class="s1">'"$http_referer" "$http_user_agent"'</span><span class="p">;</span>
access_log /var/log/nginx/access-special.log specialLog<span class="p">;</span>
access_log /var/log/nginx/access.log<span class="p">;</span>
error_log /var/log/nginx/error.log<span class="p">;</span>
<span class="c"># ...</span>
<span class="o">}</span></code></pre></figure>
<ul>
<li>Or we can override the default log format.</li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">http <span class="o">{</span>
<span class="c"># ...</span>
<span class="c">##</span>
<span class="c"># Logging Settings</span>
<span class="c">##</span>
log_format combined <span class="s1">'$http_x_real_ip - $remote_user [$time_local] '</span>
<span class="s1">'"$request" $status $body_bytes_sent '</span>
<span class="s1">'"$http_referer" "$http_user_agent"'</span><span class="p">;</span>
access_log /var/log/nginx/access.log combined<span class="p">;</span>
error_log /var/log/nginx/error.log<span class="p">;</span>
<span class="c"># ...</span>
<span class="o">}</span></code></pre></figure>
<h2 id="be-careful">Be careful</h2>
<p>In some cases you will need to add this configuration :</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">set_real_ip_from x.x.x.x/x<span class="p">;</span> <span class="c"># Ip/network of the reverse proxy (or ip received into REMOTE_ADDR)</span>
real_ip_header X-Forwarded-For<span class="p">;</span></code></pre></figure>
<h1 id="resources">Resources</h1>
<ul>
<li><a href="https://easyengine.io/tutorials/nginx/forwarding-visitors-real-ip/">https://easyengine.io/tutorials/nginx/forwarding-visitors-real-ip/</a></li>
</ul>
<p><a href="https://ypereirareis.github.io/blog/2017/02/15/nginx-real-ip-behind-nginx-reverse-proxy/">Get user real ip in nginx behind nginx reverse proxy</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on February 15, 2017.</p>
https://ypereirareis.github.io/blog/2017/02/14/nginx-redirect-http-to-https-non-www-to-www2017-02-14T00:00:00+00:002017-02-14T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p><img src="/images/posts/nginx.gif" alt="Docker" /></p>
<h1 id="redirect-http-to-https-httpwwwexamplecom-to-httpsexamplecom">Redirect HTTP to HTTPS (http://www.example.com to https://example.com)</h1>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">server <span class="o">{</span>
server_name www.example.com<span class="p">;</span>
listen 80 <span class="p">;</span>
access_log /var/log/nginx/access.log vhost<span class="p">;</span>
<span class="k">return </span>301 https://<span class="nv">$host$request_uri</span><span class="p">;</span>
<span class="o">}</span></code></pre></figure>
<h1 id="redirect-non-www-to-www-httpsexamplecom-to-httpswwwexamplecom">Redirect non-WWW to WWW (https://example.com to https://www.example.com)</h1>
<p><strong>SSL certificate configuration must be defined</strong></p>
<ul>
<li>ssl_certificate /etc/nginx/certs/example.com.crt;</li>
<li>ssl_certificate_key /etc/nginx/certs/example.com.key;</li>
<li>ssl_dhparam /etc/nginx/certs/example.com.dhparam.pem;</li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">server <span class="o">{</span>
server_name example.com<span class="p">;</span>
listen 443 ssl http2 <span class="p">;</span>
ssl_certificate /etc/nginx/certs/example.com.crt<span class="p">;</span>
ssl_certificate_key /etc/nginx/certs/example.com.key<span class="p">;</span>
ssl_dhparam /etc/nginx/certs/example.com.dhparam.pem<span class="p">;</span>
<span class="k">return </span>301 <span class="nv">$scheme</span>://www.example.com<span class="nv">$request_uri</span><span class="p">;</span>
<span class="o">}</span></code></pre></figure>
<h1 id="redirect-httpexamplecom-to-httpswwwexamplecom">Redirect http://example.com to https://www.example.com</h1>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">server <span class="o">{</span>
listen 80<span class="p">;</span>
server_name example.com<span class="p">;</span>
<span class="k">return </span>301 https://www.example.com<span class="nv">$request_uri</span><span class="p">;</span>
<span class="o">}</span></code></pre></figure>
<p>We could have merged this configuration with the first one :</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">server <span class="o">{</span>
listen 80 <span class="p">;</span>
server_name example.com www.example.com<span class="p">;</span>
<span class="k">return </span>301 https://<span class="nv">$server_name$request_uri</span><span class="p">;</span>
<span class="o">}</span></code></pre></figure>
<h1 id="redirect-ip-address-to-domain-name">Redirect IP address to domain name</h1>
<p><strong>Of course your SSL certificate must be valid for the IP address</strong></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">server <span class="o">{</span>
listen 80<span class="p">;</span>
server_name xxx.xxx.xxx.xxx<span class="p">;</span>
<span class="k">return </span>301 https://example.com<span class="nv">$request_uri</span><span class="p">;</span>
<span class="o">}</span>
server <span class="o">{</span>
server_name xxx.xxx.xxx.xxx<span class="p">;</span>
listen 443 ssl http2 <span class="p">;</span>
ssl_certificate /etc/nginx/certs/example.com.crt<span class="p">;</span>
ssl_certificate_key /etc/nginx/certs/example.com.key<span class="p">;</span>
ssl_dhparam /etc/nginx/certs/example.com.dhparam.pem<span class="p">;</span>
<span class="k">return </span>301 https://example.com<span class="nv">$request_uri</span><span class="p">;</span>
<span class="o">}</span></code></pre></figure>
<h1 id="main-config-catching-httpswwwexamplecom-used-as-a-reverse-proxy-here">Main config catching https://www.example.com used as a reverse proxy here</h1>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">server <span class="o">{</span>
server_name www.example.com<span class="p">;</span>
listen 443 ssl http2 <span class="p">;</span>
access_log /var/log/nginx/access.log vhost<span class="p">;</span>
ssl_protocols TLSv1 TLSv1.1 TLSv1.2<span class="p">;</span>
ssl_ciphers <span class="s1">'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS'</span><span class="p">;</span>
ssl_prefer_server_ciphers on<span class="p">;</span>
ssl_session_timeout 5m<span class="p">;</span>
ssl_session_cache shared:SSL:50m<span class="p">;</span>
ssl_session_tickets off<span class="p">;</span>
ssl_certificate /etc/nginx/certs/www.example.com.crt<span class="p">;</span>
ssl_certificate_key /etc/nginx/certs/www.example.com.key<span class="p">;</span>
ssl_dhparam /etc/nginx/certs/www.example.com.dhparam.pem<span class="p">;</span>
add_header Strict-Transport-Security <span class="s2">"max-age=31536000"</span><span class="p">;</span>
include /etc/nginx/vhost.d/default<span class="p">;</span>
location / <span class="o">{</span>
proxy_pass http://www.example.com<span class="p">;</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></figure>
<p><a href="https://ypereirareis.github.io/blog/2017/02/14/nginx-redirect-http-to-https-non-www-to-www/">Nginx redirect http to https and non www to www</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on February 14, 2017.</p>
https://ypereirareis.github.io/blog/2016/09/28/docker-how-to-write-good-entry-point2016-09-28T00:00:00+00:002016-09-28T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p><img src="/images/posts/docker.gif" alt="Docker" /></p>
<h1 id="a-good-docker-entry-point">A good Docker entry point</h1>
<p>To create a good docker entry point for your image or your container, you must take care of a simple thing.
<strong>The entry point script must not fail if you run it twice or more.</strong></p>
<p>Indeed, in a container the data is persisted. If you stop and restart a container without removing it,
the state/data does not change and can leads to errros.</p>
<h2 id="a-bad-entry-point-example">A bad entry point example.</h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mv /tmp/nginx/conf.d/default /etc/nginx/conf.d/default
nginx &
</code></pre></div></div>
<h2 id="what-is-the-problem-">What is the problem ?</h2>
<ul>
<li>The first time the container is started, everything works fine.</li>
<li>The second time the file <code class="language-plaintext highlighter-rouge">/tmp/nginx/conf.d/default</code> no more exists and the <code class="language-plaintext highlighter-rouge">mv</code> command generates an error.</li>
</ul>
<p>To bypass the problem, you can remove the container with <code class="language-plaintext highlighter-rouge">docker rm -f container_id|name</code> and restart it.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Test your entry points by executing them twice, at least…</p>
<p><a href="https://ypereirareis.github.io/blog/2016/09/28/docker-how-to-write-good-entry-point/">How to write a good docker entry point for docker images and containers</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on September 28, 2016.</p>
https://ypereirareis.github.io/blog/2016/09/19/ssh-tunnel-local-remote-port-forwarding2016-09-19T00:00:00+00:002016-09-19T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p><img src="/images/posts/openssh.gif" alt="open ssh" /></p>
<p>To use SSH tunnels, of course you need to have a few things:</p>
<ul>
<li>A server accessible from your local machine/network through SSH (port 22 by default) and from internet (80, 443,…).</li>
<li>SSH service running on the remote server and on your local machine.</li>
<li>An SSH key on your local machine allowed (authorized_keys) on the remote accessible server for a user.</li>
</ul>
<p>If you want to access to a remote service (BDD, intranet, …) running on a remote server
or on a machine in a remote private network… you need to use <strong>Local Port Forwarding</strong>.</p>
<h1 id="local-port-forwarding">Local Port Forwarding</h1>
<p><strong>Service directly on the remote server</strong></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">ssh <span class="nt">-Ng</span> <span class="nt">-p</span> 54322 <span class="nt">-i</span> ~/.ssh/topsecret <span class="nt">-L</span> 3320:127.0.0.1:3306 user@remoteserver.fr</code></pre></figure>
<ul>
<li><strong>-p 54322</strong> because the default ssh port has been changed on the remote server.</li>
<li><strong>-i ~/.ssh/topsecret</strong> to use the ssh key named “topsecret”.</li>
<li><strong>-L</strong> for Local port forwarding.</li>
<li><strong>3320</strong>:127.0.0.1:3306 because we forward local port 3320 on the remote machine.</li>
<li>3320:<strong>127.0.0.1</strong>:3306 because we forward the 3320 local port directly on the remote machine.</li>
<li>3320:127.0.0.1:<strong>3306</strong> because the local port if forwarded on the 3306 port on the remote machine.</li>
<li><strong>user@remoteserver.fr</strong> user and remote server domain for ssh connection.</li>
</ul>
<p><strong>Service on the private network of the remote server</strong></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">ssh <span class="nt">-Ng</span> <span class="nt">-p</span> 54322 <span class="nt">-i</span> ~/.ssh/topsecret <span class="nt">-L</span> 3320:10.1.1.54:3306 user@remoteserver.fr</code></pre></figure>
<ul>
<li>3320:<strong>10.1.1.54</strong>:3306 because we want to access port 3306 of the machine with IP 10.1.1.54 on the remote private network.
It means that the remote server must be part of the private network or be allowed to access it.</li>
</ul>
<h1 id="remote-port-forwarding">Remote Port Forwarding</h1>
<p><strong>Service directly on the local machine</strong></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">ssh <span class="nt">-nNT</span> <span class="nt">-i</span> ~/.ssh/id_rsa <span class="nt">-R</span> 9000:localhost:80 user@remoteserver.fr</code></pre></figure>
<ul>
<li><strong>9000</strong>:localhost:80 because we use port 9000 of the remote server.</li>
<li>9000:<strong>localhost</strong>:80 because we want to access to the local machine.</li>
<li>9000:localhost:<strong>80</strong> because we want to access to port 80 of the local machine.</li>
<li><strong>-i ~/.ssh/id_rsa</strong> to use the ssh key named “id_rsa”.</li>
<li><strong>user@remoteserver.fr</strong> user and remote server domain for ssh connection.</li>
</ul>
<p><strong>Service on the private network of the local machine</strong></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">ssh <span class="nt">-nNT</span> <span class="nt">-i</span> ~/.ssh/id_rsa <span class="nt">-R</span> 9000:192.168.0.88:80 user@remoteserver.fr</code></pre></figure>
<ul>
<li>9000:<strong>192.168.0.88</strong>:80 because we want to access port 80 of the machine with IP 192.168.0.88 on the local private network.
It means that the local machine must be part of the private LAN or be allowed to access it (NAT).</li>
</ul>
<h1 id="dynamic-port-forwarding">Dynamic Port Forwarding</h1>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">ssh <span class="nt">-nNT</span> <span class="nt">-C</span> <span class="nt">-D</span> 1080 user@remoteserver.fr</code></pre></figure>
<ul>
<li>-D <strong>1080</strong> because we want to use port 1080 as the dynamic port.</li>
</ul>
<p>To try this Dynamic Port Forwarding configuration just execute this command:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">curl <span class="nt">--proxy</span> socks5h://localhost:1080 http://www.my-ip-address.net/fr</code></pre></figure>
<p>Or configure your browser correctly:</p>
<ul>
<li><a href="http://sockslist.net/articles/socks-firefox-how-to">http://sockslist.net/articles/socks-firefox-how-to</a></li>
</ul>
<h1 id="use-case">USE CASE</h1>
<p>I very often use <strong>Local Port Forwarding</strong> to connect to my databases in production.
I use docker and docker-compose and it’s useful for me to expose my databases connections.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">db:
image: mysql:5.6
ports:
- 127.0.0.1:3306:3306
...</code></pre></figure>
<p><em>The goal is to expose port 3306 only on IP 127.0.0.1 and not to the entire world (0.0.0.0 by default).</em></p>
<p>Then I start a <strong>Local Port Forwarding</strong> with <code class="language-plaintext highlighter-rouge">ssh -Ng -i ~/.ssh/topsecret -L 4406:127.0.0.1:3306 user@remoteserver.fr</code>,
and I use <code class="language-plaintext highlighter-rouge">mysql -hremoteserver.fr -P 4406</code> or MySqlWorkbench over SSH.</p>
<p><a href="https://ypereirareis.github.io/blog/2016/09/19/ssh-tunnel-local-remote-port-forwarding/">Local, remote and dynamic port forwarding with SSH tunnels</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on September 19, 2016.</p>
https://ypereirareis.github.io/blog/2016/05/12/docker-machine-generic-driver-devicemapper2016-05-12T00:00:00+00:002016-05-12T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p><a href="https://docs.docker.com/machine/">docker-machine</a> is a really powerful tool to control your remote docker daemons.
Let’s see how to configure everything to manage your remote containers easily from your local host.</p>
<p><img src="/images/posts/docker.gif" alt="Docker" /></p>
<h1 id="remote-server">Remote server</h1>
<h2 id="docker-machine-user">Docker machine user</h2>
<p>Add a user on the remote server to control your docker daemon:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo </span>adduser dockeradmin</code></pre></figure>
<p>Create a custom ssh key and add it on the remote server
to allow connection with this user.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">ssh-keygen <span class="nt">-t</span> rsa <span class="nt">-b</span> 2048
ssh-copy-id <span class="nt">-i</span> ~/.ssh/dockeradmin.pem <span class="o">[</span><span class="nt">-p</span> 22345] dockeradmin@domain.fr</code></pre></figure>
<h2 id="sudo-or-not-sudo">Sudo or not sudo</h2>
<p>Your user must have <code class="language-plaintext highlighter-rouge">sudo</code> access without asking for password:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>nano /etc/sudoers
<span class="c"># User alias specification</span>
dockeradmin <span class="nv">ALL</span><span class="o">=(</span>ALL<span class="o">)</span> NOPASSWD: ALL
dockeradmin <span class="nv">ALL</span><span class="o">=(</span>ALL<span class="o">)</span> NOPASSWD: /bin/netstat</code></pre></figure>
<h2 id="netstat">Netstat</h2>
<p>Your user must have <code class="language-plaintext highlighter-rouge">netstat</code> access.
As I’m using a <strong>grs</strong> kernel I need to create a wrapper to add <code class="language-plaintext highlighter-rouge">netstat</code> access
for the dockeradmin user:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">cat </span>netstat
<span class="c">#!/bin/bash</span>
<span class="nb">exec</span> /usr/bin/sudo /bin/netstat <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="nv">$ </span><span class="nb">chmod</span> +x netstat
<span class="nv">$ </span><span class="nb">sudo cp </span>netstat /usr/local/bin/</code></pre></figure>
<h2 id="iptables">Iptables</h2>
<p>By default docker-machine uses port <code class="language-plaintext highlighter-rouge">2376</code> to communicate with docker daemons.
Of course we need to open this port on the remote server:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Docker machine port 2376</span>
iptables <span class="nt">-t</span> filter <span class="nt">-A</span> INPUT <span class="nt">-p</span> tcp <span class="nt">--dport</span> 2376 <span class="nt">-j</span> ACCEPT
iptables <span class="nt">-t</span> filter <span class="nt">-A</span> OUTPUT <span class="nt">-p</span> tcp <span class="nt">--dport</span> 2376 <span class="nt">-j</span> ACCEPT</code></pre></figure>
<h2 id="docker-daemon">Docker daemon</h2>
<p><code class="language-plaintext highlighter-rouge">docker-machine</code> and <code class="language-plaintext highlighter-rouge">generic</code> driver do not work with <code class="language-plaintext highlighter-rouge">aufs</code> storage driver.
So we need to explicitly define the <code class="language-plaintext highlighter-rouge">storage-driver</code> as <code class="language-plaintext highlighter-rouge">devicemapper</code>
on the server daemon side and on the docker-machine client.</p>
<p>On my remote server, my processes are managed by <code class="language-plaintext highlighter-rouge">systemd</code>,
a part of this configuration is automatically updated by the docker-machine client:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo cat</span> /etc/systemd/system/docker.service
<span class="o">[</span>Service]
<span class="nv">ExecStart</span><span class="o">=</span>/usr/bin/docker daemon <span class="nt">-H</span> tcp://0.0.0.0:2376 <span class="nt">-H</span> unix:///var/run/docker.sock <span class="nt">--storage-driver</span> devicemapper <span class="nt">--tlsverify</span> <span class="nt">--tlscacert</span> /etc/docker/ca.pem <span class="nt">--tlscert</span> /etc/docker/server.pem <span class="nt">--tlskey</span> /etc/docker/server-key.pem <span class="nt">--label</span> <span class="nv">provider</span><span class="o">=</span>generic
<span class="nv">MountFlags</span><span class="o">=</span>slave
<span class="nv">LimitNOFILE</span><span class="o">=</span>1048576
<span class="nv">LimitNPROC</span><span class="o">=</span>1048576
<span class="nv">LimitCORE</span><span class="o">=</span>infinity
<span class="nv">Environment</span><span class="o">=</span>
<span class="o">[</span>Install]
<span class="nv">WantedBy</span><span class="o">=</span>multi-user.target</code></pre></figure>
<h1 id="local-configuration">Local configuration</h1>
<h2 id="install-docker-machine">Install docker-machine</h2>
<p><a href="https://docs.docker.com/machine/install-machine/">https://docs.docker.com/machine/install-machine/</a></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>docker-machine version
docker-machine version 0.6.0, build e27fb87</code></pre></figure>
<h2 id="create-your-first-machine">Create your first machine</h2>
<p>The command to start your <code class="language-plaintext highlighter-rouge">docker-machine</code> is the following. Note the use of specific:</p>
<ul>
<li>ssh port</li>
<li>ssk key</li>
</ul>
<p><strong>Very important,</strong> the <code class="language-plaintext highlighter-rouge">--engine-storage-driver devicemapper</code> configuration:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker-machine create <span class="nt">-d</span> generic <span class="se">\</span>
<span class="nt">--generic-ssh-user</span> dockeradmin <span class="se">\</span>
<span class="nt">--generic-ssh-key</span> ~/.ssh/dockeradmin.pem <span class="se">\</span>
<span class="nt">--generic-ssh-port</span> 22XXX <span class="se">\</span>
<span class="nt">--engine-storage-driver</span> devicemapper <span class="se">\</span>
<span class="nt">--generic-ip-address</span> domain.fr <span class="se">\</span>
MACHINE_NAME
<span class="nv">$ </span>docker-machine <span class="nb">ls
</span>NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
MACHINE_NAME - generic Running tcp://domain.fr:2376 v1.11.0 </code></pre></figure>
<h2 id="switch-between-environments">Switch between environments</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>docker-machine <span class="nb">env </span>MACHINE_NAME
<span class="nb">export </span><span class="nv">DOCKER_TLS_VERIFY</span><span class="o">=</span><span class="s2">"1"</span>
<span class="nb">export </span><span class="nv">DOCKER_HOST</span><span class="o">=</span><span class="s2">"tcp://domain.fr:2376"</span>
<span class="nb">export </span><span class="nv">DOCKER_CERT_PATH</span><span class="o">=</span><span class="s2">"/home/USER/.docker/machine/machines/MACHINE_NAME"</span>
<span class="nb">export </span><span class="nv">DOCKER_MACHINE_NAME</span><span class="o">=</span><span class="s2">"MACHINE_NAME"</span>
<span class="c"># Run this command to configure your shell: </span>
<span class="nv">$ </span><span class="nb">eval</span> <span class="si">$(</span>docker-machine <span class="nb">env </span>MACHINE_NAME<span class="si">)</span></code></pre></figure>
<p>Execute a <code class="language-plaintext highlighter-rouge">docker ps</code> and you will control your remote daemon and see your remote containers.</p>
<p>Reset your client configuration to manage your local containers.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">eval</span> <span class="si">$(</span>docker-machine <span class="nb">env</span> <span class="nt">--unset</span><span class="si">)</span></code></pre></figure>
<h1 id="sources">Sources</h1>
<ul>
<li><a href="http://www.thegeekstuff.com/2016/02/docker-machine-create-generic/">http://www.thegeekstuff.com/2016/02/docker-machine-create-generic/</a></li>
<li><a href="https://docs.docker.com/engine/admin/systemd/">https://docs.docker.com/engine/admin/systemd/</a></li>
<li><a href="https://blog.dahanne.net/2015/10/07/adding-an-existing-docker-host-to-docker-machine-a-few-tips/">https://blog.dahanne.net/2015/10/07/adding-an-existing-docker-host-to-docker-machine-a-few-tips/</a></li>
<li><a href="https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/">https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/</a></li>
<li><a href="https://docs.docker.com/engine/admin/configuring/">https://docs.docker.com/engine/admin/configuring/</a></li>
</ul>
<p><a href="https://ypereirareis.github.io/blog/2016/05/12/docker-machine-generic-driver-devicemapper/">Use docker-machine with generic driver and devicemapper storage driver</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on May 12, 2016.</p>
https://ypereirareis.github.io/blog/2016/03/29/doctrine-parameters-type-hint-tip-mysql-index2016-03-29T00:00:00+00:002016-03-29T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p>As you may know, <a href="http://www.doctrine-project.org/">Doctrine</a> can be a very good ally, but if we do not use it correctly
we can have big performance problems really easily and quickly.</p>
<p><img src="/images/posts/doctrine.gif" alt="Symfony" /></p>
<p>Let’s take these two tables as an example:</p>
<p><strong>Players</strong></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">+------------------------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------------------------+--------------+------+-----+---------+-------+
| <span class="nb">id</span> | int<span class="o">(</span>11<span class="o">)</span> | NO | PRI | NULL | |
| name | varchar<span class="o">(</span>255<span class="o">)</span> | YES | | NULL | |
+------------------------------+--------------+------+-----+---------+-------+</code></pre></figure>
<p><strong>Players Actions</strong></p>
<p>Players can do actions, each action is defined by a name and a format.
Of course, an action is done by a single player.
A Player can do an action of a given format only once. (Not really important actually)</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">+------------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+-------------+------+-----+---------+----------------+
| <span class="nb">id</span> | int<span class="o">(</span>11<span class="o">)</span> | NO | PRI | NULL | auto_increment |
| player_id | int<span class="o">(</span>11<span class="o">)</span> | NO | MUL | NULL | |
| action | varchar<span class="o">(</span>20<span class="o">)</span> | NO | | NULL | |
| format | varchar<span class="o">(</span>20<span class="o">)</span> | NO | | NULL | |
+------------+-------------+------+-----+---------+----------------+</code></pre></figure>
<h1 id="bad-dql-example">Bad DQL example</h1>
<p>We could write this king of DQL query,
it works and it will generate a valid SQL query :</p>
<h2 id="dql-query">DQL Query</h2>
<p>In the example, let’s say that the player has id 52.</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp"><?php</span>
<span class="nv">$qb</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="na">doctrine</span><span class="o">-></span><span class="na">getRepository</span><span class="p">(</span><span class="s1">'AppBundle:ActionPlayer'</span><span class="p">)</span>
<span class="o">-></span><span class="na">createQueryBuilder</span><span class="p">(</span><span class="s1">'ap'</span><span class="p">)</span>
<span class="o">-></span><span class="na">select</span><span class="p">(</span><span class="s1">'ap.format, ap.action, COUNT(ap.player) AS value'</span><span class="p">)</span>
<span class="o">-></span><span class="na">andWhere</span><span class="p">(</span><span class="s1">'ap.player = :player'</span><span class="p">)</span><span class="o">-></span><span class="na">setParameter</span><span class="p">(</span><span class="s1">'player'</span><span class="p">,</span> <span class="nv">$player</span><span class="p">)</span>
<span class="o">-></span><span class="na">addGroupBy</span><span class="p">(</span><span class="s1">'ap.player'</span><span class="p">)</span>
<span class="o">-></span><span class="na">addGroupBy</span><span class="p">(</span><span class="s1">'ap.format'</span><span class="p">)</span>
<span class="o">-></span><span class="na">addGroupBy</span><span class="p">(</span><span class="s1">'ap.action'</span><span class="p">)</span>
<span class="p">;</span></code></pre></figure>
<h2 id="generated-sql-mysql">Generated SQL (Mysql)</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">SELECT g0_.format AS format_0, g0_.action AS action_1, COUNT<span class="o">(</span>g0_.player_id<span class="o">)</span> AS sclr_2
FROM action_player g0_
WHERE g0_.player_id <span class="o">=</span> <span class="s1">'52'</span>
GROUP BY g0_.player_id, g0_.format, g0_.action<span class="p">;</span></code></pre></figure>
<h1 id="good-dql-example-not-really-actually">Good DQL example (not really actually)</h1>
<h2 id="dql-query-1">DQL Query</h2>
<p>In the example, let’s say that the player has id 52.</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp"><?php</span>
<span class="nv">$qb</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="na">doctrine</span><span class="o">-></span><span class="na">getRepository</span><span class="p">(</span><span class="s1">'CommonBundle:Stats\StatActionPlayer'</span><span class="p">)</span>
<span class="o">-></span><span class="na">createQueryBuilder</span><span class="p">(</span><span class="s1">'ap'</span><span class="p">)</span>
<span class="o">-></span><span class="na">select</span><span class="p">(</span><span class="s1">'ap.format, ap.action, COUNT(ap.player) AS value'</span><span class="p">)</span>
<span class="o">-></span><span class="na">innerJoin</span><span class="p">(</span><span class="s1">'ap.player'</span><span class="p">,</span> <span class="s1">'player'</span><span class="p">)</span>
<span class="o">-></span><span class="na">andWhere</span><span class="p">(</span><span class="s1">'player.id = :player_id'</span><span class="p">)</span><span class="o">-></span><span class="na">setParameter</span><span class="p">(</span><span class="s1">'player_id'</span><span class="p">,</span> <span class="nv">$player</span><span class="p">)</span>
<span class="o">-></span><span class="na">addGroupBy</span><span class="p">(</span><span class="s1">'ap.player'</span><span class="p">)</span>
<span class="o">-></span><span class="na">addGroupBy</span><span class="p">(</span><span class="s1">'ap.format'</span><span class="p">)</span>
<span class="o">-></span><span class="na">addGroupBy</span><span class="p">(</span><span class="s1">'ap.action'</span><span class="p">)</span>
<span class="p">;</span></code></pre></figure>
<h2 id="generated-sql-mysql-1">Generated SQL (Mysql)</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">SELECT g0_.format AS format_0, g0_.action AS action_1, COUNT<span class="o">(</span>g0_.player_id<span class="o">)</span> AS sclr_2
FROM action_player g0_
INNER JOIN player g1_ ON g0_.player_id <span class="o">=</span> g1_.id
WHERE g1_.id <span class="o">=</span> <span class="s1">'52'</span>
GROUP BY g0_.player_id, g0_.format, g0_.action<span class="p">;</span></code></pre></figure>
<h1 id="so-whats-the-problem-">So! What’s the problem ?</h1>
<p>Let’s say we have millions of rows in our <code class="language-plaintext highlighter-rouge">action_player</code> table…
We need an index to improve SELECT performance:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">ALTER TABLE action_player ADD INDEX action_player_idx <span class="o">(</span>player_id, format, action<span class="o">)</span><span class="p">;</span></code></pre></figure>
<p><strong>Important to note that the index must contains <code class="language-plaintext highlighter-rouge">GROUP BY</code> fields AND field used in <code class="language-plaintext highlighter-rouge">WHERE</code> clause.</strong></p>
<p>Actually, if we go back to our two previous SQL queries (auto generated)
and we try to execute them on our table with millions of rows and the above index:</p>
<ul>
<li>The first one gives results in about <strong>3,9s</strong>.</li>
<li>The second one gives results in about <strong>0,36s</strong>.</li>
</ul>
<h2 id="the-difference-is-huge">The difference is HUGE….</h2>
<p>How are those two queries really executed by MySQL ?</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">mysql> EXPLAIN SELECT g0_.format AS format_0, g0_.action AS action_1, COUNT<span class="o">(</span>g0_.player_id<span class="o">)</span> AS sclr_2
-> FROM action_player g0_
-> WHERE g0_.player_id <span class="o">=</span> <span class="s1">'52'</span>
-> GROUP BY g0_.player_id, g0_.format, g0_.action<span class="p">;</span>
+----+-------------+-------+------+-----------------------------------------+----------------------+...
| <span class="nb">id</span> | select_type | table | <span class="nb">type</span> | possible_keys | key |...
+----+-------------+-------+------+-----------------------------------------+----------------------+...
| 1 | SIMPLE | g0_ | ref | IDX_4D92B9D399E6F5DF, action_player_idx | IDX_4D92B9D399E6F5DF |...
+----+-------------+-------+------+-----------------------------------------+----------------------+...
1 row <span class="k">in </span><span class="nb">set</span> <span class="o">(</span>0.00 sec<span class="o">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">mysql> EXPLAIN SELECT g0_.format AS format_0, g0_.action AS action_1, COUNT<span class="o">(</span>g0_.player_id<span class="o">)</span> AS sclr_2
-> FROM action_player g0_
-> INNER JOIN player g1_ ON g0_.player_id <span class="o">=</span> g1_.id
-> WHERE g1_.id <span class="o">=</span> <span class="s1">'52'</span>
-> GROUP BY g0_.player_id, g0_.format, g0_.action<span class="p">;</span>
+----+-------------+-------+-------+-----------------------------------------+-------------------+...
| <span class="nb">id</span> | select_type | table | <span class="nb">type</span> | possible_keys | key |...
+----+-------------+-------+-------+-----------------------------------------+-------------------+...
| 1 | SIMPLE | g1_ | const | PRIMARY | PRIMARY |...
| 1 | SIMPLE | g0_ | ref | IDX_4D92B9D399E6F5DF, action_player_idx | action_player_idx |...
+----+-------------+-------+-------+-----------------------------------------+-------------------+...
2 rows <span class="k">in </span><span class="nb">set</span> <span class="o">(</span>0.00 sec<span class="o">)</span></code></pre></figure>
<ul>
<li>In the first case the created index <strong>IS NOT</strong> used.</li>
<li>The index <strong>IS</strong> used in the second case.</li>
</ul>
<h2 id="explanation">Explanation</h2>
<ul>
<li>When an index is created, it takes into account the type of columns used in the index.</li>
<li>The player id field is an integer, so the index is created with an integer for the player_id field.</li>
<li>In the bad DQL example, Doctrine generates an SQL query with <code class="language-plaintext highlighter-rouge">WHERE g0_.player_id = '52'</code> <strong>and NO join</strong>, the index is not used !</li>
<li>In the good DQL example, Doctrine generates an SQL query with <code class="language-plaintext highlighter-rouge">WHERE g0_.player_id = '52'</code> <strong>AND a join on player table</strong>, the index is used for the inner join !</li>
</ul>
<h1 id="the-solution-improve-your-dql-queries-with-type-hinting">The solution, improve your DQL queries with type hinting.</h1>
<p>In these two examples, we can see that the generated SQL contains this part <code class="language-plaintext highlighter-rouge">WHERE g1_.id = '52'</code>.</p>
<p>Actually, THIS IS the problem, not really the join between tables.
In both DQL queries the <code class="language-plaintext highlighter-rouge">player_id</code> is used as a string but it should be as an integer.</p>
<p>Always add type hinting to your DQL parameters:</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp"><?php</span>
<span class="nv">$qb</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="na">doctrine</span><span class="o">-></span><span class="na">getRepository</span><span class="p">(</span><span class="s1">'AppBundle:ActionPlayer'</span><span class="p">)</span>
<span class="o">-></span><span class="na">createQueryBuilder</span><span class="p">(</span><span class="s1">'ap'</span><span class="p">)</span>
<span class="o">-></span><span class="na">select</span><span class="p">(</span><span class="s1">'ap.format, ap.action, COUNT(ap.player) AS value'</span><span class="p">)</span>
<span class="o">-></span><span class="na">andWhere</span><span class="p">(</span><span class="s1">'ap.player = :player'</span><span class="p">)</span><span class="o">-></span><span class="na">setParameter</span><span class="p">(</span><span class="s1">'player'</span><span class="p">,</span> <span class="p">(</span><span class="nx">int</span><span class="p">)</span> <span class="nv">$player</span><span class="p">)</span>
<span class="o">-></span><span class="na">addGroupBy</span><span class="p">(</span><span class="s1">'ap.player'</span><span class="p">)</span>
<span class="o">-></span><span class="na">addGroupBy</span><span class="p">(</span><span class="s1">'ap.format'</span><span class="p">)</span>
<span class="o">-></span><span class="na">addGroupBy</span><span class="p">(</span><span class="s1">'ap.action'</span><span class="p">)</span>
<span class="p">;</span></code></pre></figure>
<p>Or you could use the third parameter of the <code class="language-plaintext highlighter-rouge">setParameter()</code> method:</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp"><?php</span>
<span class="o">-></span><span class="na">andWhere</span><span class="p">(</span><span class="s1">'ap.player = :player'</span><span class="p">)</span><span class="o">-></span><span class="na">setParameter</span><span class="p">(</span><span class="s1">'player'</span><span class="p">,</span> <span class="nv">$player</span><span class="p">,</span> <span class="nx">\Doctrine\DBAL\Types\Type</span><span class="o">::</span><span class="na">INTEGER</span><span class="p">)</span></code></pre></figure>
<p><strong>No more join needed to filter by player_id</strong> and the index will be used as it should be.</p>
<p><a href="https://ypereirareis.github.io/blog/2016/03/29/doctrine-parameters-type-hint-tip-mysql-index/">Doctrine performance tip with MySQL and indexes, parameters type hinting</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on March 29, 2016.</p>
https://ypereirareis.github.io/blog/2016/03/22/mysql-insert-ignore-alternatives2016-03-22T00:00:00+00:002016-03-22T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p>It’s not always really easy to deal with “INSERT if not exists” operations.
Many solutions exist with MySQL:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">INSERT IGNORE INTO table VALUES (...)</code></li>
<li><code class="language-plaintext highlighter-rouge">REPLACE INTO table VALUES (...)</code></li>
<li><code class="language-plaintext highlighter-rouge">SELECT id FROM table</code> then <code class="language-plaintext highlighter-rouge">INTO table VALUES (...)</code> or <code class="language-plaintext highlighter-rouge">UPDATE table</code></li>
<li><code class="language-plaintext highlighter-rouge">INSERT INTO table SELECT id FROM table WHERE NOT EXISTS</code></li>
</ul>
<p><img src="/images/posts/mysql.jpg" alt="Symfony" /></p>
<h1 id="problems-with-the-three-first-solutions">Problems with the three first solutions</h1>
<h2 id="insert-ignore">Insert ignore</h2>
<p>Problems:</p>
<ul>
<li>No more errors triggered, but it’s what we want.</li>
<li>Auto increment IDs are incremented, even if a given record already exists.</li>
</ul>
<h2 id="replace-into">Replace into</h2>
<p>Problems:</p>
<ul>
<li>Auto increment IDs are incremented, even if a given record already exists.</li>
<li>This operation actually does a <code class="language-plaintext highlighter-rouge">DELETE</code> then an <code class="language-plaintext highlighter-rouge">INSERT</code>.</li>
<li>If <code class="language-plaintext highlighter-rouge">DELETE CASCADE</code> configured, you’re gonna blew up your database.</li>
</ul>
<h2 id="select-then-insert-or-replace">Select then insert or replace</h2>
<p>Problems:</p>
<ul>
<li>Many SQL queries.</li>
<li>Many requests over the network.</li>
<li>Less performance (execution time) than previous solutions.</li>
</ul>
<h1 id="insert-into-select-where-not-exists">Insert into select where not exists</h1>
<p>It’s for me the best solution, but not always fits your needs:</p>
<ul>
<li>No ID consumption if value already exists.</li>
<li>No unwanted delete in case of <code class="language-plaintext highlighter-rouge">DELETE CASCADE</code> configuration.</li>
<li>A single simple mysql request.</li>
</ul>
<p>A fully working example:</p>
<figure class="highlight"><pre><code class="language-mysql" data-lang="mysql">INSERT INTO table_name (firstname, lastname)
SELECT 'NEW FIRSTNAME', 'NEW LASTNAME'
FROM DUAL
WHERE NOT EXISTS(
SELECT 1
FROM table_name
WHERE firstname = 'NEW FIRSTNAME' AND lastname = 'NEW LASTNAME'
)
LIMIT 1;</code></pre></figure>
<p><strong>Maybe, you will need to add an index on the fields you use to filter the sub-query.</strong></p>
<h2 id="what-is-this-from-dual-">What is this FROM DUAL ?</h2>
<p>Actually we could have use <code class="language-plaintext highlighter-rouge">FROM table_name</code> like that:</p>
<figure class="highlight"><pre><code class="language-mysql" data-lang="mysql">INSERT INTO table_name ...
SELECT ...
FROM table_name
WHERE NOT EXISTS(
...
)
LIMIT 1;</code></pre></figure>
<p><strong>But there is a difference, if the table is empty this SQL query will never add anything in the table.</strong></p>
<h3 id="yes-what-is-this-from-dual-">Yes…. what is this <code class="language-plaintext highlighter-rouge">FROM DUAL</code> ?</h3>
<p>Just read the <a href="http://dev.mysql.com/doc/refman/5.7/en/select.html">MySQL documentation</a> for <code class="language-plaintext highlighter-rouge">SELECT FROM DUAL</code>
or the <a href="https://en.wikipedia.org/wiki/DUAL_table">Wikipedia definition</a>.</p>
<blockquote>
<p>DUAL is purely for the convenience of people who require that all SELECT statements should have FROM and possibly other clauses.
MySQL may ignore the clauses. MySQL does not require FROM DUAL if no tables are referenced.</p>
</blockquote>
<p>If you need to add another clause like <code class="language-plaintext highlighter-rouge">WHERE 1=1</code> for the example, you will must use <code class="language-plaintext highlighter-rouge">FROM DUAL</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">mysql> <span class="k">select </span>1+1<span class="p">;</span>
+-----+
| 1+1 |
+-----+
| 2 |
+-----+
1 row <span class="k">in </span><span class="nb">set</span> <span class="o">(</span>0.00 sec<span class="o">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">mysql> <span class="k">select </span>1+1 WHERE <span class="nv">1</span><span class="o">=</span>1<span class="p">;</span>
ERROR 1064 <span class="o">(</span>42000<span class="o">)</span>: You have an error <span class="k">in </span>your SQL syntax<span class="p">;</span> check the manual that corresponds to your MySQL server version <span class="k">for </span>the right syntax to use near <span class="s1">'WHERE 1=1'</span> at line 1</code></pre></figure>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">mysql> <span class="k">select </span>1+1 FROM DUAL WHERE <span class="nv">1</span><span class="o">=</span>1<span class="p">;</span>
+-----+
| 1+1 |
+-----+
| 2 |
+-----+
1 row <span class="k">in </span><span class="nb">set</span> <span class="o">(</span>0.00 sec<span class="o">)</span></code></pre></figure>
<p><a href="https://ypereirareis.github.io/blog/2016/03/22/mysql-insert-ignore-alternatives/">MySQL INSERT IGNORE alternatives</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on March 22, 2016.</p>
https://ypereirareis.github.io/blog/2016/03/16/symfony-lexikjwtauthenticationbundle-client-user-authenticator-provider2016-03-16T00:00:00+00:002016-03-16T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p>We can find on the internet many examples on how to implements a client consuming API authenticated with JWT.
But impossible to find one based on Symfony and how to configure firewall, user provider, user authenticator and a user model.
So let’s see how to do it.</p>
<p><img src="/images/posts/symfony.gif" alt="Symfony" /></p>
<h1 id="lexikjwtauthenticationbundle-server-side">LexikJWTAuthenticationBundle Server side</h1>
<p>To create an API secured with LexikJWTAuthenticationBundle and JWT,
just read and follow step by step <a href="https://github.com/lexik/LexikJWTAuthenticationBundle/blob/master/Resources/doc/index.md">the documentation</a>.
Everything should be fine as the doc is pretty good.</p>
<p>As we use custom roles for our users we need to expose them in the JWT token.
Indeed, we need to add them in our users on the client side.</p>
<h2 id="user-roles-in-the-jwt-token">User roles in the JWT token</h2>
<p>Just create a JWT listener:</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">AdminBundle\Security</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Lexik\Bundle\JWTAuthenticationBundle\Event\JWTCreatedEvent</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">JMS\DiExtraBundle\Annotation</span> <span class="k">as</span> <span class="n">DI</span><span class="p">;</span>
<span class="cd">/**
* Class JWTCreatedListener
* @package AdminBundle\Security
*
* @DI\Service("project.listener.jwt_created")
* @DI\Tag("kernel.event_listener", attributes = {
* "event" = "lexik_jwt_authentication.on_jwt_created", "method": "onJWTCreated"
* })
*
*/</span>
<span class="kd">class</span> <span class="nc">JWTCreatedListener</span>
<span class="p">{</span>
<span class="cd">/**
* @param JWTCreatedEvent $event
*
* @return void
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">onJWTCreated</span><span class="p">(</span><span class="nx">JWTCreatedEvent</span> <span class="nv">$event</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="p">(</span><span class="nv">$request</span> <span class="o">=</span> <span class="nv">$event</span><span class="o">-></span><span class="na">getRequest</span><span class="p">()))</span> <span class="p">{</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="nv">$user</span> <span class="o">=</span> <span class="nv">$event</span><span class="o">-></span><span class="na">getUser</span><span class="p">();</span>
<span class="nv">$payload</span> <span class="o">=</span> <span class="nv">$event</span><span class="o">-></span><span class="na">getData</span><span class="p">();</span>
<span class="nv">$payload</span><span class="p">[</span><span class="s1">'roles'</span><span class="p">]</span> <span class="o">=</span> <span class="nv">$user</span><span class="o">-></span><span class="na">getRoles</span><span class="p">();</span>
<span class="nv">$event</span><span class="o">-></span><span class="na">setData</span><span class="p">(</span><span class="nv">$payload</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<h2 id="auth-from-query-param">Auth from query param</h2>
<p>Sometimes you will need to generate forms on the server side, then serialize them into a json response, to be able to show them in the front.</p>
<p>The form action on submission will point to the server side, and it could be difficult to add the JWT token in the headers.</p>
<p>So I advise you to allow JWT authentication with query param.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">security:
firewalls:
api:
pattern: ^/api
provider: fos_userbundle <span class="c"># or something else</span>
stateless: <span class="nb">true
</span>anonymous: <span class="nb">true
</span>lexik_jwt:
query_parameter:
enabled: <span class="nb">true
</span>name: bearer <span class="c"># or something else</span></code></pre></figure>
<p>Of course when generating your forms views before exposing them to the API, do not forget to add the JWT token as a query string param.</p>
<figure class="highlight"><pre><code class="language-html" data-lang="html"><span class="nt"><form</span>
<span class="na">method=</span><span class="s">"POST"</span>
<span class="na">action=</span><span class="s">"{{ url('your_route', {bearer: jwt_token}) }}"</span>
<span class="err">{{</span> <span class="na">form_enctype</span><span class="err">(</span><span class="na">form</span><span class="err">)</span> <span class="err">}}</span>
<span class="nt">></span>
...
<span class="nt"></form></span></code></pre></figure>
<h1 id="symfony-jwt-client-authenticatorprovider">Symfony JWT client (authenticator/provider)</h1>
<p>As examples are better than words… let’s configurations examples.</p>
<h2 id="security">Security</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">security:
providers:
token:
<span class="nb">id</span>: project.token.user_provider
firewalls:
dev:
pattern: ^/<span class="o">(</span>_<span class="o">(</span>profiler|wdt<span class="o">)</span>|css|images|js<span class="o">)</span>/
security: <span class="nb">false
</span>main:
pattern: ^/
provider: token
anonymous: <span class="nb">true
</span>simple_form:
authenticator: project.token.authenticator
check_path: login_check
login_path: login
use_referer: <span class="nb">true
</span>failure_path: login
<span class="nb">logout</span>:
path: /logout
target: login
remember_me:
secret: <span class="s1">'%secret%'</span>
lifetime: 86400
path: /
access_control:
- <span class="o">{</span> path: ^/login, role: IS_AUTHENTICATED_ANONYMOUSLY <span class="o">}</span>
- <span class="o">{</span> path: ^/editor, role: ROLE_EDITOR <span class="o">}</span>
- <span class="o">{</span> path: ^/registration, role: IS_AUTHENTICATED_ANONYMOUSLY <span class="o">}</span>
- <span class="o">{</span> path: ^/, role: IS_AUTHENTICATED_ANONYMOUSLY <span class="o">}</span></code></pre></figure>
<h2 id="routing">Routing</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">login_check:
pattern: /secured/login_check
<span class="nb">logout</span>:
path: /logout</code></pre></figure>
<h2 id="user-model">User model</h2>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">AppBundle\Security</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\Security\Core\User\AdvancedUserInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\Security\Core\User\UserInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\Security\Core\User\EquatableInterface</span><span class="p">;</span>
<span class="kd">class</span> <span class="nc">ApiUser</span> <span class="k">implements</span> <span class="nx">AdvancedUserInterface</span><span class="p">,</span> <span class="nx">\Serializable</span><span class="p">,</span> <span class="nx">EquatableInterface</span>
<span class="p">{</span>
<span class="k">private</span> <span class="nv">$username</span><span class="p">;</span>
<span class="k">private</span> <span class="nv">$password</span><span class="p">;</span>
<span class="k">private</span> <span class="nv">$salt</span><span class="p">;</span>
<span class="k">private</span> <span class="nv">$roles</span><span class="p">;</span>
<span class="k">private</span> <span class="nv">$token</span><span class="p">;</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">__construct</span><span class="p">(</span><span class="nv">$username</span><span class="p">,</span> <span class="nv">$password</span><span class="p">,</span> <span class="nv">$salt</span><span class="p">,</span> <span class="k">array</span> <span class="nv">$roles</span><span class="p">,</span> <span class="nv">$token</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">username</span> <span class="o">=</span> <span class="nv">$username</span><span class="p">;</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">password</span> <span class="o">=</span> <span class="nv">$password</span><span class="p">;</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">salt</span> <span class="o">=</span> <span class="nv">$salt</span><span class="p">;</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">roles</span> <span class="o">=</span> <span class="nv">$roles</span><span class="p">;</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">token</span> <span class="o">=</span> <span class="nv">$token</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">getRoles</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nv">$this</span><span class="o">-></span><span class="na">roles</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">getPassword</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nv">$this</span><span class="o">-></span><span class="na">password</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">getSalt</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nv">$this</span><span class="o">-></span><span class="na">salt</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">getUsername</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nv">$this</span><span class="o">-></span><span class="na">username</span><span class="p">;</span>
<span class="p">}</span>
<span class="cd">/**
* @return mixed
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">getToken</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nv">$this</span><span class="o">-></span><span class="na">token</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">eraseCredentials</span><span class="p">()</span>
<span class="p">{</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">isEqualTo</span><span class="p">(</span><span class="nx">UserInterface</span> <span class="nv">$user</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nv">$user</span> <span class="nx">instanceof</span> <span class="nx">self</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="kc">false</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="na">password</span> <span class="o">!==</span> <span class="nv">$user</span><span class="o">-></span><span class="na">getPassword</span><span class="p">())</span> <span class="p">{</span>
<span class="k">return</span> <span class="kc">false</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="na">salt</span> <span class="o">!==</span> <span class="nv">$user</span><span class="o">-></span><span class="na">getSalt</span><span class="p">())</span> <span class="p">{</span>
<span class="k">return</span> <span class="kc">false</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="na">username</span> <span class="o">!==</span> <span class="nv">$user</span><span class="o">-></span><span class="na">getUsername</span><span class="p">())</span> <span class="p">{</span>
<span class="k">return</span> <span class="kc">false</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">return</span> <span class="kc">true</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">isAccountNonExpired</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="kc">true</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">isAccountNonLocked</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="kc">true</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">isCredentialsNonExpired</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="kc">true</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">isEnabled</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="kc">true</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">serialize</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nb">serialize</span><span class="p">([</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">token</span><span class="p">,</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">username</span><span class="p">,</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">password</span><span class="p">,</span>
<span class="p">]);</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">unserialize</span><span class="p">(</span><span class="nv">$serialized</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">list</span> <span class="p">(</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">token</span><span class="p">,</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">username</span><span class="p">,</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">password</span><span class="p">,</span>
<span class="p">)</span> <span class="o">=</span> <span class="nb">unserialize</span><span class="p">(</span><span class="nv">$serialized</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<h2 id="authenticator">Authenticator</h2>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">AppBundle\Security</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">AppBundle\Repository\RepositoryInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Psr\Log\LoggerInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\HttpFoundation\Request</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\HttpFoundation\Response</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\HttpKernel\Exception\HttpException</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\Security\Core\Authentication\Token\TokenInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\Security\Core\Authentication\Token\UsernamePasswordToken</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\Security\Core\Exception\AuthenticationException</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\Security\Core\Exception\CustomUserMessageAuthenticationException</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\Security\Core\User\UserProviderInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\Security\Http\Authentication\SimpleFormAuthenticatorInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">JMS\DiExtraBundle\Annotation</span> <span class="k">as</span> <span class="n">DI</span><span class="p">;</span>
<span class="cd">/**
* Token Authenticator.
*
* @DI\Service("project.token.authenticator")
*/</span>
<span class="kd">class</span> <span class="nc">TokenAuthenticator</span> <span class="k">implements</span> <span class="nx">SimpleFormAuthenticatorInterface</span>
<span class="p">{</span>
<span class="cd">/**
* @var RepositoryInterface
*/</span>
<span class="k">protected</span> <span class="nv">$repository</span><span class="p">;</span>
<span class="cd">/**
* @var LoggerInterface
*/</span>
<span class="k">protected</span> <span class="nv">$logger</span><span class="p">;</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">authenticateToken</span><span class="p">(</span><span class="nx">TokenInterface</span> <span class="nv">$token</span><span class="p">,</span> <span class="nx">UserProviderInterface</span> <span class="nv">$userProvider</span><span class="p">,</span> <span class="nv">$providerKey</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">try</span> <span class="p">{</span>
<span class="nv">$user</span> <span class="o">=</span> <span class="nv">$token</span><span class="o">-></span><span class="na">getUser</span><span class="p">();</span>
<span class="nv">$userProvider</span><span class="o">-></span><span class="na">getUsernameForApiKey</span><span class="p">(</span><span class="nv">$user</span><span class="o">-></span><span class="na">getToken</span><span class="p">());</span>
<span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">\Exception</span> <span class="nv">$e</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// CAUTION: this message will be returned to the client</span>
<span class="c1">// (so don't put any un-trusted messages / error strings here)</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nx">CustomUserMessageAuthenticationException</span><span class="p">(</span><span class="s1">'Invalid username or password'</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="k">new</span> <span class="nx">UsernamePasswordToken</span><span class="p">(</span>
<span class="nv">$user</span><span class="p">,</span>
<span class="nv">$user</span><span class="o">-></span><span class="na">getPassword</span><span class="p">(),</span>
<span class="nv">$providerKey</span><span class="p">,</span>
<span class="nv">$user</span><span class="o">-></span><span class="na">getRoles</span><span class="p">()</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">supportsToken</span><span class="p">(</span><span class="nx">TokenInterface</span> <span class="nv">$token</span><span class="p">,</span> <span class="nv">$providerKey</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nv">$token</span> <span class="nx">instanceof</span> <span class="nx">UsernamePasswordToken</span>
<span class="o">&&</span> <span class="nv">$token</span><span class="o">-></span><span class="na">getProviderKey</span><span class="p">()</span> <span class="o">===</span> <span class="nv">$providerKey</span><span class="p">;</span>
<span class="p">}</span>
<span class="cd">/**
* TokenAuthenticator constructor.
*
* @param RepositoryInterface $repository
*
* @DI\InjectParams({
* "repository" = @DI\Inject("project.repository.api"),
* })
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">__construct</span><span class="p">(</span><span class="nx">LoggerInterface</span> <span class="nv">$logger</span><span class="p">,</span> <span class="nx">RepositoryInterface</span> <span class="nv">$repository</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">logger</span> <span class="o">=</span> <span class="nv">$logger</span><span class="p">;</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">repository</span> <span class="o">=</span> <span class="nv">$repository</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">createToken</span><span class="p">(</span><span class="nx">Request</span> <span class="nv">$request</span><span class="p">,</span> <span class="nv">$username</span><span class="p">,</span> <span class="nv">$password</span><span class="p">,</span> <span class="nv">$providerKey</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">try</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="kc">null</span> <span class="o">===</span> <span class="nv">$username</span> <span class="o">||</span> <span class="kc">null</span> <span class="o">===</span> <span class="nv">$password</span><span class="p">)</span> <span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nx">AuthenticationException</span><span class="p">(</span><span class="s1">'Username and password must be defined'</span><span class="p">);</span>
<span class="p">}</span>
<span class="nv">$data</span> <span class="o">=</span> <span class="p">[</span>
<span class="s1">'form_params'</span> <span class="o">=></span> <span class="p">[</span>
<span class="s1">'_username'</span> <span class="o">=></span> <span class="nv">$username</span><span class="p">,</span>
<span class="s1">'_password'</span> <span class="o">=></span> <span class="nv">$password</span><span class="p">,</span>
<span class="p">],</span>
<span class="p">];</span>
<span class="k">try</span> <span class="p">{</span>
<span class="c1">// Call here your server to get a JWT Token from username and password.</span>
<span class="c1">// I Use an API Repository based on Guzzle.</span>
<span class="nv">$clientResponse</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="na">repository</span><span class="o">-></span><span class="na">loginCheck</span><span class="p">(</span><span class="nv">$data</span><span class="p">);</span>
<span class="nv">$token</span> <span class="o">=</span> <span class="nb">json_decode</span><span class="p">(</span><span class="nv">$clientResponse</span><span class="o">-></span><span class="na">getBody</span><span class="p">(),</span> <span class="kc">true</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nb">isset</span><span class="p">(</span><span class="nv">$token</span><span class="p">[</span><span class="s1">'token'</span><span class="p">]))</span> <span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nx">AuthenticationException</span><span class="p">(</span><span class="s1">'API No Auth Token returned'</span><span class="p">);</span>
<span class="p">}</span>
<span class="nv">$apiKey</span> <span class="o">=</span> <span class="nv">$token</span><span class="p">[</span><span class="s1">'token'</span><span class="p">];</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nv">$apiKey</span><span class="p">)</span> <span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nx">AuthenticationException</span><span class="p">(</span><span class="s1">'API No Key found'</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">list</span><span class="p">(</span><span class="nv">$username</span><span class="p">,</span> <span class="nv">$roles</span><span class="p">)</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="na">getUsernameForApiKey</span><span class="p">(</span><span class="nv">$apiKey</span><span class="p">);</span>
<span class="nv">$user</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">ApiUser</span><span class="p">(</span><span class="nv">$username</span><span class="p">,</span> <span class="nv">$password</span><span class="p">,</span> <span class="s1">''</span><span class="p">,</span> <span class="nv">$roles</span><span class="p">,</span> <span class="nv">$apiKey</span><span class="p">);</span>
<span class="k">return</span> <span class="k">new</span> <span class="nx">UsernamePasswordToken</span><span class="p">(</span>
<span class="nv">$user</span><span class="p">,</span>
<span class="nv">$password</span><span class="p">,</span>
<span class="nv">$providerKey</span><span class="p">,</span>
<span class="nv">$roles</span>
<span class="p">);</span>
<span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">HttpException</span> <span class="nv">$ex</span><span class="p">)</span> <span class="p">{</span>
<span class="k">switch</span> <span class="p">(</span><span class="nv">$ex</span><span class="o">-></span><span class="na">getStatusCode</span><span class="p">())</span> <span class="p">{</span>
<span class="k">case</span> <span class="nx">Response</span><span class="o">::</span><span class="na">HTTP_UNAUTHORIZED</span><span class="o">:</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nx">AuthenticationException</span><span class="p">(</span><span class="s1">'API Unauthorized: '</span><span class="o">.</span> <span class="nv">$ex</span><span class="o">-></span><span class="na">getMessage</span><span class="p">());</span>
<span class="k">case</span> <span class="nx">Response</span><span class="o">::</span><span class="na">HTTP_FORBIDDEN</span><span class="o">:</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nx">AuthenticationException</span><span class="p">(</span><span class="s1">'API Forbidden: '</span><span class="o">.</span> <span class="nv">$ex</span><span class="o">-></span><span class="na">getMessage</span><span class="p">());</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">AuthenticationException</span> <span class="nv">$ex</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">logger</span><span class="o">-></span><span class="na">error</span><span class="p">(</span><span class="nv">$ex</span><span class="o">-></span><span class="na">getMessage</span><span class="p">());</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nx">CustomUserMessageAuthenticationException</span><span class="p">(</span><span class="s1">'Invalid username or password'</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<h2 id="provider">Provider</h2>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">AppBundle\Security</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Psr\Log\LoggerInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\Security\Core\Exception\AuthenticationException</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\Security\Core\Exception\CustomUserMessageAuthenticationException</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\Security\Core\User\UserProviderInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\Security\Core\User\User</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\Security\Core\User\UserInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\Security\Core\Exception\UnsupportedUserException</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">JMS\DiExtraBundle\Annotation</span> <span class="k">as</span> <span class="n">DI</span><span class="p">;</span>
<span class="cd">/**
* Token User Provider.
*
* @DI\Service("project.token.user_provider")
*/</span>
<span class="kd">class</span> <span class="nc">TokenUserProvider</span> <span class="k">implements</span> <span class="nx">UserProviderInterface</span>
<span class="p">{</span>
<span class="k">const</span> <span class="no">JWT_TOKEN_PARTS_COUNT</span> <span class="o">=</span> <span class="mi">3</span><span class="p">;</span>
<span class="k">const</span> <span class="no">TOKEN_REFRESH_DELAY</span> <span class="o">=</span> <span class="mi">120</span><span class="p">;</span>
<span class="cd">/**
* TokenUserProvider constructor.
*
* @param LoggerInterface $logger
*
*
* @DI\InjectParams({
* })
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">__construct</span><span class="p">(</span><span class="nx">LoggerInterface</span> <span class="nv">$logger</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">logger</span> <span class="o">=</span> <span class="nv">$logger</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">getUsernameForApiKey</span><span class="p">(</span><span class="nv">$apiKey</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">try</span> <span class="p">{</span>
<span class="nv">$tokenParts</span> <span class="o">=</span> <span class="nb">explode</span><span class="p">(</span><span class="s1">'.'</span><span class="p">,</span> <span class="nv">$apiKey</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">self</span><span class="o">::</span><span class="na">JWT_TOKEN_PARTS_COUNT</span> <span class="o">!==</span> <span class="nb">count</span><span class="p">(</span><span class="nv">$tokenParts</span><span class="p">))</span> <span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nx">AuthenticationException</span><span class="p">(</span><span class="s1">'TOKEN Wrong Auth Token format'</span><span class="p">);</span>
<span class="p">}</span>
<span class="nv">$payload</span> <span class="o">=</span> <span class="nb">json_decode</span><span class="p">(</span><span class="nb">base64_decode</span><span class="p">(</span><span class="nv">$tokenParts</span><span class="p">[</span><span class="mi">1</span><span class="p">]),</span> <span class="kc">true</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nb">isset</span><span class="p">(</span><span class="nv">$payload</span><span class="p">[</span><span class="s1">'username'</span><span class="p">]))</span> <span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nx">AuthenticationException</span><span class="p">(</span><span class="s1">'TOKEN No Username found in the Auth Token'</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nb">isset</span><span class="p">(</span><span class="nv">$payload</span><span class="p">[</span><span class="s1">'exp'</span><span class="p">]))</span> <span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nx">AuthenticationException</span><span class="p">(</span><span class="s1">'TOKEN No expiration timestamp found in the Auth Token'</span><span class="p">);</span>
<span class="p">}</span>
<span class="nv">$roles</span> <span class="o">=</span> <span class="nb">isset</span><span class="p">(</span><span class="nv">$payload</span><span class="p">[</span><span class="s1">'roles'</span><span class="p">])</span> <span class="o">?</span> <span class="nv">$payload</span><span class="p">[</span><span class="s1">'roles'</span><span class="p">]</span> <span class="o">:</span> <span class="p">[];</span>
<span class="nv">$exp</span> <span class="o">=</span> <span class="nv">$payload</span><span class="p">[</span><span class="s1">'exp'</span><span class="p">];</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$exp</span> <span class="o">+</span> <span class="p">(</span><span class="nx">int</span><span class="p">)</span> <span class="nx">self</span><span class="o">::</span><span class="na">TOKEN_REFRESH_DELAY</span> <span class="o"><=</span> <span class="nb">time</span><span class="p">())</span> <span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nx">AuthenticationException</span><span class="p">(</span><span class="s1">'TOKEN Expired'</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="p">[</span>
<span class="nv">$payload</span><span class="p">[</span><span class="s1">'username'</span><span class="p">],</span>
<span class="nv">$roles</span>
<span class="p">];</span>
<span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">\Exception</span> <span class="nv">$ex</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">logger</span><span class="o">-></span><span class="na">error</span><span class="p">(</span><span class="nv">$ex</span><span class="o">-></span><span class="na">getMessage</span><span class="p">());</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nx">CustomUserMessageAuthenticationException</span><span class="p">(</span><span class="s1">'You have been disconnected, try to reconnect.'</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">loadUserByUsername</span><span class="p">(</span><span class="nv">$username</span><span class="p">)</span>
<span class="p">{</span>
<span class="c1">// NOT USED IN OUR CASE !!!</span>
<span class="k">return</span> <span class="k">new</span> <span class="nx">ApiUser</span><span class="p">(</span><span class="nv">$username</span><span class="p">,</span> <span class="kc">null</span><span class="p">,</span> <span class="s1">''</span><span class="p">,</span> <span class="p">[</span><span class="s1">'ROLE_USER'</span><span class="p">],</span> <span class="s1">''</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">refreshUser</span><span class="p">(</span><span class="nx">UserInterface</span> <span class="nv">$user</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nv">$user</span> <span class="nx">instanceof</span> <span class="nx">ApiUser</span><span class="p">)</span> <span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nx">UnsupportedUserException</span><span class="p">(</span>
<span class="nb">sprintf</span><span class="p">(</span><span class="s1">'Instances of "%s" are not supported.'</span><span class="p">,</span> <span class="nb">get_class</span><span class="p">(</span><span class="nv">$user</span><span class="p">))</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="k">list</span><span class="p">(</span><span class="nv">$username</span><span class="p">,</span> <span class="nv">$roles</span><span class="p">)</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="na">getUsernameForApiKey</span><span class="p">(</span><span class="nv">$user</span><span class="o">-></span><span class="na">getToken</span><span class="p">());</span>
<span class="k">return</span> <span class="k">new</span> <span class="nx">ApiUser</span><span class="p">(</span><span class="nv">$username</span><span class="p">,</span> <span class="kc">null</span><span class="p">,</span> <span class="s1">''</span><span class="p">,</span> <span class="nv">$roles</span><span class="p">,</span> <span class="nv">$user</span><span class="o">-></span><span class="na">getToken</span><span class="p">());</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">supportsClass</span><span class="p">(</span><span class="nv">$class</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="s1">'AppBundle\Security\ApiUser'</span> <span class="o">===</span> <span class="nv">$class</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<h2 id="authenticated-api-calls"> Authenticated API calls</h2>
<p>Actually you can get your user as usual and get JWT token stored inside the user model.
Let’s see an example of an “API repository”</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">AppBundle\Repository\Api</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">AppBundle\Repository\RepositoryInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">AppBundle\Security\ApiUser</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">GuzzleHttp\Client</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">GuzzleHttp\Exception\RequestException</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">JMS\DiExtraBundle\Annotation</span> <span class="k">as</span> <span class="n">DI</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Psr\Log\LoggerInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\HttpKernel\Exception\HttpException</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\HttpKernel\KernelInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\Security\Core\Authentication\Token\Storage\TokenStorageInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nn">Symfony\Component\Security\Core\Security</span><span class="p">;</span>
<span class="cd">/**
* Class BaseRepository.
*
* @DI\Service("project.repository.api", abstract=true)
*/</span>
<span class="k">abstract</span> <span class="kd">class</span> <span class="nc">BaseRepository</span> <span class="k">implements</span> <span class="nx">RepositoryInterface</span>
<span class="p">{</span>
<span class="cd">/**
* @var ClientRegistry
*/</span>
<span class="k">protected</span> <span class="nv">$client</span><span class="p">;</span>
<span class="cd">/**
* @var KernelInterface
*/</span>
<span class="k">protected</span> <span class="nv">$kernel</span><span class="p">;</span>
<span class="cd">/**
* @var LoggerInterface
*/</span>
<span class="k">protected</span> <span class="nv">$logger</span><span class="p">;</span>
<span class="cd">/**
* @var TokenStorageInterface
*/</span>
<span class="k">protected</span> <span class="nv">$securityTokenStorage</span><span class="p">;</span>
<span class="cd">/**
* BaseRepository constructor.
* @param KernelInterface $kernel
* @param LoggerInterface $logger
* @param ClientRegistry $client
* @param TokenStorageInterface $securityTokenStorage
*
* @DI\InjectParams({
* "client" = @DI\Inject("project.registry.client"),
* "securityTokenStorage" = @DI\Inject("security.token_storage"),
* })
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">__construct</span><span class="p">(</span><span class="nx">KernelInterface</span> <span class="nv">$kernel</span><span class="p">,</span> <span class="nx">LoggerInterface</span> <span class="nv">$logger</span><span class="p">,</span> <span class="nx">ClientRegistry</span> <span class="nv">$client</span><span class="p">,</span> <span class="nx">TokenStorageInterface</span> <span class="nv">$securityTokenStorage</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">kernel</span> <span class="o">=</span> <span class="nv">$kernel</span><span class="p">;</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">logger</span> <span class="o">=</span> <span class="nv">$logger</span><span class="p">;</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">client</span> <span class="o">=</span> <span class="nv">$client</span><span class="p">;</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">securityTokenStorage</span> <span class="o">=</span> <span class="nv">$securityTokenStorage</span><span class="p">;</span>
<span class="p">}</span>
<span class="cd">/**
* @param $url
* @param bool $public
* @return mixed
*/</span>
<span class="k">protected</span> <span class="k">function</span> <span class="nf">getData</span><span class="p">(</span><span class="nv">$url</span><span class="p">,</span> <span class="nv">$public</span> <span class="o">=</span> <span class="kc">true</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">try</span> <span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">logger</span><span class="o">-></span><span class="na">debug</span><span class="p">(</span><span class="s1">'API call with Guzzle'</span><span class="p">,</span> <span class="p">[</span><span class="s1">'url'</span><span class="p">,</span> <span class="nv">$url</span><span class="p">]);</span>
<span class="nv">$client</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="na">client</span><span class="o">-></span><span class="na">get</span><span class="p">();</span>
<span class="nv">$options</span> <span class="o">=</span> <span class="p">[];</span>
<span class="nv">$token</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="na">getUserToken</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(</span><span class="kc">null</span> <span class="o">!==</span> <span class="nv">$token</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$options</span> <span class="o">=</span> <span class="nb">array_merge_recursive</span><span class="p">(</span>
<span class="nv">$options</span><span class="p">,</span> <span class="p">[</span>
<span class="s1">'headers'</span> <span class="o">=></span> <span class="p">[</span>
<span class="s1">'Authorization'</span> <span class="o">=></span> <span class="nb">sprintf</span><span class="p">(</span><span class="s1">'Bearer %s'</span><span class="p">,</span> <span class="nv">$token</span><span class="p">),</span>
<span class="p">],</span>
<span class="p">]);</span>
<span class="nv">$url</span> <span class="o">.=</span> <span class="nb">sprintf</span><span class="p">(</span><span class="s1">'?bearer=%s'</span><span class="p">,</span> <span class="nv">$token</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nv">$client</span><span class="o">-></span><span class="na">get</span><span class="p">(</span><span class="nv">$url</span><span class="p">,</span> <span class="nv">$options</span><span class="p">);</span>
<span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">RequestException</span> <span class="nv">$ex</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$response</span> <span class="o">=</span> <span class="nv">$ex</span><span class="o">-></span><span class="na">getResponse</span><span class="p">();</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nx">HttpException</span><span class="p">(</span><span class="nv">$response</span><span class="o">-></span><span class="na">getStatusCode</span><span class="p">(),</span> <span class="nv">$ex</span><span class="o">-></span><span class="na">getMessage</span><span class="p">()</span><span class="o">.</span><span class="s1">'-'</span><span class="o">.</span><span class="nv">$response</span><span class="o">-></span><span class="na">getReasonPhrase</span><span class="p">());</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">protected</span> <span class="k">function</span> <span class="nf">getUserToken</span><span class="p">()</span>
<span class="p">{</span>
<span class="nv">$user</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="na">securityTokenStorage</span><span class="o">-></span><span class="na">getToken</span><span class="p">()</span><span class="o">-></span><span class="na">getUser</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(</span><span class="nb">is_object</span><span class="p">(</span><span class="nv">$user</span><span class="p">)</span> <span class="o">&&</span> <span class="nv">$user</span> <span class="nx">instanceof</span> <span class="nx">ApiUser</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nv">$user</span><span class="o">-></span><span class="na">getToken</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">return</span> <span class="kc">null</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<h2 id="conclusion">Conclusion</h2>
<p>This configuration has been used as a POC. Feel free to change or optimize it.
Feedback appreciated too !</p>
<p><a href="https://ypereirareis.github.io/blog/2016/03/16/symfony-lexikjwtauthenticationbundle-client-user-authenticator-provider/">Symfony client for an API secured with LexikJWTAuthenticationBundle, user authenticator and user provider</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on March 16, 2016.</p>
https://ypereirareis.github.io/blog/2016/03/11/symfony-doctrine-transactional-code-made-easy2016-03-11T00:00:00+00:002016-03-11T00:00:00+00:00Yannick Pereira-Reishttps://ypereirareis.github.io
<p>When doing multiple database operations in a single http request, command line, method,…
we often need to use a database transaction to keep data safe.</p>
<p><img src="/images/posts/symfony.gif" alt="Symfony" /></p>
<h1 id="example-in-an-action-of-a-controller">Example in an Action of a Controller</h1>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp"><?php</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">testAction</span><span class="p">()</span>
<span class="p">{</span>
<span class="nv">$conn</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="na">getDoctrine</span><span class="p">()</span><span class="o">-></span><span class="na">getConnection</span><span class="p">();</span>
<span class="nv">$conn</span><span class="o">-></span><span class="na">setAutoCommit</span><span class="p">(</span><span class="kc">false</span><span class="p">);</span>
<span class="nv">$conn</span><span class="o">-></span><span class="na">beginTransaction</span><span class="p">();</span>
<span class="k">try</span> <span class="p">{</span>
<span class="nv">$everythingIsFine</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="na">get</span><span class="p">(</span><span class="s1">'service'</span><span class="p">)</span><span class="o">-></span><span class="na">do</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$everythingIsFine</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$conn</span><span class="o">-></span><span class="na">commit</span><span class="p">();</span>
<span class="k">return</span> <span class="k">new</span> <span class="nx">Response</span><span class="p">(</span><span class="s2">"OK"</span><span class="p">);</span>
<span class="p">}</span>
<span class="nv">$conn</span><span class="o">-></span><span class="na">rollback</span><span class="p">();</span>
<span class="k">return</span> <span class="k">new</span> <span class="nx">Response</span><span class="p">(</span><span class="s2">"NOT OK"</span><span class="p">);</span>
<span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">\Exception</span> <span class="nv">$ex</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$conn</span><span class="o">-></span><span class="na">rollback</span><span class="p">();</span>
<span class="k">return</span> <span class="k">new</span> <span class="nx">Response</span><span class="p">(</span><span class="s2">"NOT OK"</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<h1 id="a-better-choice">A better choice</h1>
<ul>
<li>Add a method that allows you to keep your code DRY.</li>
<li>This method gets a <code class="language-plaintext highlighter-rouge">callable</code> param (a function/method to execute) and deals with potential exceptions, and transaction <code class="language-plaintext highlighter-rouge">commit()</code> and <code class="language-plaintext highlighter-rouge">rollback()</code> operations.</li>
<li>It’s a really dead simple example and a reminder, feel free to improve it.</li>
<li>You could also move it into a service for instance.</li>
</ul>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp"><?php</span>
<span class="k">protected</span> <span class="k">function</span> <span class="nf">transactionalExec</span><span class="p">(</span><span class="nx">callable</span> <span class="nv">$func</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$conn</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="na">getDoctrine</span><span class="p">()</span><span class="o">-></span><span class="na">getConnection</span><span class="p">();</span>
<span class="nv">$conn</span><span class="o">-></span><span class="na">setAutoCommit</span><span class="p">(</span><span class="kc">false</span><span class="p">);</span>
<span class="nv">$conn</span><span class="o">-></span><span class="na">beginTransaction</span><span class="p">();</span>
<span class="k">try</span> <span class="p">{</span>
<span class="nv">$success</span> <span class="o">=</span> <span class="nv">$func</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(</span><span class="kc">null</span> <span class="o">===</span> <span class="nv">$success</span><span class="p">)</span> <span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nx">\Exception</span><span class="p">(</span><span class="s1">'Your transactional callable must return a boolean value'</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$success</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$conn</span><span class="o">-></span><span class="na">commit</span><span class="p">();</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nv">$conn</span><span class="o">-></span><span class="na">rollback</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">\Exception</span> <span class="nv">$ex</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$conn</span><span class="o">-></span><span class="na">rollback</span><span class="p">();</span>
<span class="nv">$success</span> <span class="o">=</span> <span class="kc">false</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nv">$success</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">testAction</span><span class="p">()</span>
<span class="p">{</span>
<span class="nv">$transactionalSuccess</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="na">transactionalExec</span><span class="p">(</span><span class="k">function</span><span class="p">()</span>
<span class="k">use</span> <span class="p">(</span><span class="o">...</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$everythingIsFine</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="na">get</span><span class="p">(</span><span class="s1">'service'</span><span class="p">)</span><span class="o">-></span><span class="na">do</span><span class="p">();</span>
<span class="k">return</span> <span class="nv">$everythingIsFine</span><span class="p">;</span>
<span class="p">});</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$transactionalSuccess</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nx">Response</span><span class="p">(</span><span class="s2">"OK"</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="k">new</span> <span class="nx">Response</span><span class="p">(</span><span class="s2">"NOT OK"</span><span class="p">);</span>
<span class="p">}</span></code></pre></figure>
<p><a href="https://ypereirareis.github.io/blog/2016/03/11/symfony-doctrine-transactional-code-made-easy/">How to encapsulate your doctrine operations into transaction really easily</a> was originally published by Yannick Pereira-Reis at <a href="https://ypereirareis.github.io">Yannick Pereira-Reis</a> on March 11, 2016.</p>