Jekyll2022-12-01T09:14:29+00:00https://tech.kinksters.dating/feed.xmlMeet Kinksters TechConnecting kinky people with great technology.Brad Jonesbrad@kinksters.datingImplementing the Matrix JS SDK in React Native2022-10-22T00:00:00+00:002022-10-22T00:00:00+00:00https://tech.kinksters.dating/posts/matrix-js-sdk-react-native<p>Meet Kinksters’ chat feature is backed by <a href="https://matrix.org">Matrix</a>, an awesome open-source communication protocol
and its <a href="https://github.com/matrix-org/synapse/">Synapse</a> homeserver reference implementation.</p>
<p>In our React Native app, we connect with the excellent <a href="https://github.com/matrix-org/matrix-js-sdk"><code class="language-plaintext highlighter-rouge">matrix-js-sdk</code></a>
library (which is also behind their official Element client).</p>
<p>All of the Matrix ecosystem is under rapid development but is production-ready. When using the SDK on React Native,
however, there are a few gotchas worth documenting to save others some “WTF” moments. It’s worth noting that none of
these pain points are the fault of the SDK maintainers but stem instead from limitations and implementation details
in React Native itself.</p>
<h2 id="incomplete-intl-support-in-react-native-hermes-runtime">Incomplete <code class="language-plaintext highlighter-rouge">Intl</code> support in React Native Hermes runtime</h2>
<p><a href="">Hermes</a> is a new-ish JavaScript engine for React Native that yields performance and DX benefits. However, it lacks
<a href="https://github.com/facebook/hermes/issues/23#issuecomment-1156832485">support for the <code class="language-plaintext highlighter-rouge">Intl</code> spec on iOS</a> in a few
key areas. Due to <a href="https://github.com/matrix-org/matrix-js-sdk/pull/1801">this change</a> in <code class="language-plaintext highlighter-rouge">matrix-js-sdk</code>, the Metro
bundler needs to find <code class="language-plaintext highlighter-rouge">Intl.Collator</code>, which it can’t on iOS.</p>
<p>There are various code snippets around the web on pollyfill-ing <code class="language-plaintext highlighter-rouge">Intl</code> in RN, but many of these solutions do not
support <code class="language-plaintext highlighter-rouge">Collator</code>. I spent more time than I should have trying to implement various elegant fixes, however given the
library’s narrow (single) use of this feature, I settled on the only solution that seemed to work for me - including a
homegrown polyfill.</p>
<h4 id="metroconfigjs"><code class="language-plaintext highlighter-rouge">metro.config.js</code></h4>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Learn more https://docs.expo.io/guides/customizing-metro</span>
<span class="kd">const</span> <span class="p">{</span> <span class="nx">getDefaultConfig</span> <span class="p">}</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">expo/metro-config</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">config</span> <span class="o">=</span> <span class="nx">getDefaultConfig</span><span class="p">(</span><span class="nx">__dirname</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">originalPolyfills</span> <span class="o">=</span> <span class="nx">config</span><span class="p">.</span><span class="nx">serializer</span><span class="p">.</span><span class="nx">getPolyfills</span><span class="p">;</span>
<span class="nx">config</span><span class="p">.</span><span class="nx">serializer</span><span class="p">.</span><span class="nx">getPolyfills</span> <span class="o">=</span> <span class="p">({</span><span class="nx">platform</span><span class="p">})</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">let</span> <span class="nx">polyfills</span> <span class="o">=</span> <span class="nx">originalPolyfills</span><span class="p">({</span><span class="nx">platform</span><span class="p">});</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">platform</span> <span class="o">===</span> <span class="dl">'</span><span class="s1">ios</span><span class="dl">'</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">polyfills</span> <span class="o">=</span> <span class="nx">polyfills</span><span class="p">.</span><span class="nx">concat</span><span class="p">([</span><span class="nx">__dirname</span> <span class="o">+</span> <span class="dl">'</span><span class="s1">/intl.polyfill.js</span><span class="dl">'</span><span class="p">]);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nx">polyfills</span><span class="p">;</span>
<span class="p">};</span>
<span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="nx">config</span><span class="p">;</span>
</code></pre></div></div>
<h4 id="intlpolyfilljs"><code class="language-plaintext highlighter-rouge">intl.polyfill.js</code></h4>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="p">(</span><span class="k">typeof</span> <span class="nb">global</span><span class="p">.</span><span class="nx">Intl</span> <span class="o">===</span> <span class="dl">'</span><span class="s1">undefined</span><span class="dl">'</span><span class="p">)</span> <span class="p">{</span>
<span class="nb">global</span><span class="p">.</span><span class="nx">Intl</span> <span class="o">=</span> <span class="p">{};</span>
<span class="nb">global</span><span class="p">.</span><span class="nx">Intl</span><span class="p">.</span><span class="nx">Collator</span> <span class="o">=</span> <span class="kd">function</span> <span class="p">()</span> <span class="p">{}</span>
<span class="nb">global</span><span class="p">.</span><span class="nx">Intl</span><span class="p">.</span><span class="nx">Collator</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">compare</span> <span class="o">=</span> <span class="p">(</span><span class="nx">a</span><span class="p">,</span> <span class="nx">b</span><span class="p">)</span> <span class="o">=></span> <span class="nx">a</span><span class="p">.</span><span class="nx">localeCompare</span><span class="p">(</span><span class="nx">b</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>My comfort with this patch is buoyed by the fact React Native
<a href="https://github.com/facebook/hermes/issues/23#issuecomment-1211768104"><code class="language-plaintext highlighter-rouge">0.70.0</code></a> includes the missing support. Expo
hasn’t yet rolled an SDK including this version, but I can rip this out when they do.</p>
<h2 id="missing-webcrypto-support-in-react-native">Missing <code class="language-plaintext highlighter-rouge">WebCrypto</code> support in React Native</h2>
<p><a href="https://github.com/matrix-org/matrix-js-sdk/pull/2762">This change</a> in the SDK now finds a
<a href="https://developer.mozilla.org/en-US/docs/Web/API/Web_Crypto_API">WebCrypto</a> implementation from the browser or
Node 16+. The issue of course for React Native is that we’re neither in a browser <em>nor</em> running on Node.</p>
<p><a href="https://github.com/react-native-community/discussions-and-proposals/issues/83#issuecomment-456110449">Support for WebCrypto API in React Native seems unlikely to land officially</a>
and the one polyfill I have found <a href="https://github.com/LinusU/react-native-webcrypto">only supports Android</a>. So,
this is a limitation to consider if implementing E2E encryption on RN with Matrix.</p>
<p>Our current chat implementation is not end-to-end encrypted (in order to facilitate moderation; this might change
in the future) so this isn’t a major restriction, however the SDK and Metro bundler still expect to find some <code class="language-plaintext highlighter-rouge">crypto</code>
functionality when building.</p>
<p>The same author of the full polyfill has, however, provided an
<a href="https://github.com/LinusU/react-native-get-random-values">actively-maintained polyfill</a> for just
<code class="language-plaintext highlighter-rouge">crypto.getRandomValues</code>, which is enough to satisfy Metro.</p>
<p>I’ve opened <a href="https://github.com/matrix-org/matrix-js-sdk/issues/2788">an issue</a> to perhaps more gracefully handle a
missing WebCrypto API.</p>
<h4 id="metroconfigjs-1"><code class="language-plaintext highlighter-rouge">metro.config.js</code></h4>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Learn more https://docs.expo.io/guides/customizing-metro</span>
<span class="kd">const</span> <span class="p">{</span> <span class="nx">getDefaultConfig</span> <span class="p">}</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">expo/metro-config</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="p">{</span> <span class="nx">resolver</span> <span class="p">}</span> <span class="o">=</span> <span class="nx">getDefaultConfig</span><span class="p">(</span><span class="nx">__dirname</span><span class="p">);</span>
<span class="nx">config</span><span class="p">.</span><span class="nx">resolver</span> <span class="o">=</span> <span class="p">{</span>
<span class="p">...</span><span class="nx">resolver</span><span class="p">,</span>
<span class="na">extraNodeModules</span><span class="p">:</span> <span class="p">{</span><span class="na">crypto</span><span class="p">:</span> <span class="nx">require</span><span class="p">.</span><span class="nx">resolve</span><span class="p">(</span><span class="dl">'</span><span class="s1">react-native-get-random-values</span><span class="dl">'</span><span class="p">)},</span>
<span class="p">};</span>
<span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="nx">config</span><span class="p">;</span>
</code></pre></div></div>
<h2 id="url-is-broken-and-wont-be-fixed-in-react-native"><code class="language-plaintext highlighter-rouge">URL</code> is broken and won’t be fixed in React Native</h2>
<p>Just a few days ago, the SDK updated its HTTP client from <code class="language-plaintext highlighter-rouge">browser-request</code> to <code class="language-plaintext highlighter-rouge">fetch</code>. This is super helpful to me,
since <code class="language-plaintext highlighter-rouge">request</code> is unmaintained and lacked proper middleware support. So, I was having to implement a homebrew polyfill
that delegated to <code class="language-plaintext highlighter-rouge">fetch</code> under the hood. I knew it was janky but lacked the expertise and time to provide an upstream
refactor.</p>
<p>The awesome people at Matrix did just that, and it <a href="https://github.com/matrix-org/matrix-js-sdk/pull/2719">landed just a few days ago</a>.
This did mean some non-trivial refactoring on my side, but I was happy to do so as it meant reducing my technical debt
and being able to more easily keep pace with their upstream changes.</p>
<p>After making what I thought were all the required changes, I found that Matrix was generating URLs with trailing slashes
added, such as <code class="language-plaintext highlighter-rouge">/_matrix/client/versions/</code> to which Synapse would reply with <code class="language-plaintext highlighter-rouge">{errcode: "M_UNRECOGNIZED", ...}</code>.</p>
<p>It took a bit of sleuthing to find the culprit, which was <code class="language-plaintext highlighter-rouge">URL()</code>. Turns out it’s
<a href="https://github.com/facebook/react-native/issues/24428#issuecomment-634730129">broken in React Native</a> and in such a
way that <a href="https://github.com/facebook/react-native/pull/25719#issuecomment-521286173">fixing would be a BC break</a> so
the semi-official answer is to use a polyfill, forever.</p>
<p>TL;dr: if you have an RN project, include
<a href="https://github.com/charpeni/react-native-url-polyfill"><code class="language-plaintext highlighter-rouge">react-native-url-polyfill</code></a> as a matter of course to avoid
chasing your tail.</p>Brad Jonesbrad@kinksters.datingMeet Kinksters’ chat feature is backed by Matrix, an awesome open-source communication protocol and its Synapse homeserver reference implementation.Accessing iOS console/syslog entries on Linux2022-10-09T00:00:00+00:002022-10-09T00:00:00+00:00https://tech.kinksters.dating/posts/ios-console-logs-linux<p>I’m preparing Meet Kinksters for a beta release, and a big (and time consuming, and frustrating…) part of this is
building and debugging binaries for both iOS and Android. I use the excellent and powerful Expo framework for our
React Native app and distribution. Expo is great, however it can’t ease all the pain in debugging especially when
building “production” releases which lack most development logging and debugging tools.</p>
<p>One particular frustration was debugging an almost immediate crash of my production app as soon as the splash screen
appeared. This being a production build, there is no logging to a console on my remote host/laptop. Even stranger still,
because I was running the app through Fastlane I had no crash logs in the normal Analytics settings tab.</p>
<p>Out of despiration, I started searching for ways to stream console logs from my iPad (bought specifically to test my
iOS app - under protest!) and most answers were to use Xcode. I think Apple’s closed development ecosystem is a
shameless cash grab, so no Mac for me.</p>
<p>I did however stumble upon <a href="https://docs.libimobiledevice.org/libimobiledevice/latest/"><code class="language-plaintext highlighter-rouge">libimobiledevice</code></a>, a library
with companion CLI tools to do just this!</p>
<p>The documentation is pretty sparse, but I was able to connect to my iPad and sift through the logs (they come fast and
furious) to encounter my error. Turns out I missed some configuration options when bootstrapping the Facebook Login SDK.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>idevice_id <span class="nt">--list</span>
aa4adeaf1dd2688e1c5154d00f411724e914858d
<span class="nv">$ </span>idevicesyslog <span class="nt">-u</span> aa4adeaf1dd2688e1c5154d00f411724e914858d
...
Oct 9 04:51:39 MeetKinksters<span class="o">(</span>CoreFoundation<span class="o">)[</span>12673] <Notice>: <span class="k">***</span> Terminating app due to uncaught exception <span class="s1">'NSInternalInconsistencyException'</span>, reason: <span class="s1">'Starting with v13 of the SDK, a client token must be embedded in your client code before making Graph API calls.
Visit https://developers.facebook.com/apps/2630480557258506/settings/advanced/ to find your client token for this app.
Add a key named FacebookClientToken to your Info.plist, and add your client token as its value.
Visit https://developers.facebook.com/docs/ios/getting-started#configure-your-project for more information.'</span>
<span class="k">***</span> First throw call stack:
<span class="o">(</span>0x181ab1d1c 0x1992d6ee4 0x101282ad8 0x1012825dc 0x10127ded0 0x10127e998 0x10127d080 0x10124b394 0x10124675c 0x101249494 0x181a482e8 0x181add8e4 0x181ab2c04 0x181a5d070 0x18314d04c 0x1840114c0 0x18412e05c 0x184356434 0x1842e6174 0x183ef6d44 0x183fb2934 0x183ef8ad4 0x183fdd184 0x184432f38 0x183f2677c 0x183f6851c 0x183ef4e38 0x192c1c328 0x192c3315c 0x192c19bdc 0x192c1fa68 0x181772094 0x181715150 0x192c1b2ac 0x192c1a7c0 0x192c1e960 0x181ad24fc 0x181ae262c 0x<<span class="se">\M</span><span class="nt">-b</span><span class="se">\M</span>^@<span class="se">\M</span>-&>
...
</code></pre></div></div>
<p>Just say no to Apple’s vendor lock-in!</p>Brad Jonesbrad@kinksters.datingI’m preparing Meet Kinksters for a beta release, and a big (and time consuming, and frustrating…) part of this is building and debugging binaries for both iOS and Android. I use the excellent and powerful Expo framework for our React Native app and distribution. Expo is great, however it can’t ease all the pain in debugging especially when building “production” releases which lack most development logging and debugging tools.Handling cacheability for computed Drupal entity references2022-08-13T00:00:00+00:002022-08-13T00:00:00+00:00https://tech.kinksters.dating/posts/cacheability-computed-drupal-entity-references<p>The data model for Meet Kinksters’ backend requires us to keep track of many entity relationships,
e.g. users who have shown interest in each other. Many relationships are appropriate to maintain
in a database entry, because they are integral to the entity itself (matches are a good example.)</p>
<p>Other relationships are more ephemeral, and their value is the result of some business rule
computation or introspection. One such example is enabling dating profile entities to reference
the user’s default photo. Users may change their default photo in the UI, and the reference
“out” to the photo isn’t integral to the dating profile itself. And, due to rules around
representing relationships in JSON:API
(“<a href="https://jsonapi.org/format/#document-resource-object-linkage">full linkage</a>”) sometimes
it is convenient to create and reveal these computed relationships to the API layer.</p>
<p>One of Drupal’s best features is its robust cache API;
<a href="/posts/2022-03-28-invalidating-cache-owner-drupal/">I’ve written about this previously</a>.
How can we express cache metadata about a computed entity reference field?</p>
<h2 id="setting-cache-metadata-on-a-field-item">Setting cache metadata on a field item</h2>
<p>Drupal’s developer documentation often boils down to “read the code,” and that was true
in this case. I didn’t see much by way of inspiration in any core modules, as computed
entity reference fields are a bit of a power-user feature.</p>
<p>I was aided by a <a href="https://www.drupal.org/project/drupal/issues/2997123">2021 issue and bugfix</a>
which made JSON:API module’s searializers cache aware.
<a href="https://git.drupalcode.org/project/drupal/-/commit/5ff249f5e14f938700e01772aa4607c4882b4a92#35d2a57f363a5a5fbded03ce417bd089119afeb9">The test coverage</a>
for that change provided me the example I needed to express cache metadata on a
computed field.</p>
<p>Due to the nature of computed fields, you’ll need to write custom code to ensure the
serialization respects whatever tags or cache contexts apply to your computation.</p>
<h3 id="implementing-the-cacheable-field-property">Implementing the cacheable field property</h3>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="k">declare</span><span class="p">(</span><span class="n">strict_types</span><span class="o">=</span><span class="mi">1</span><span class="p">);</span>
<span class="kn">namespace</span> <span class="nn">Drupal\my_module\Plugin\Field\FieldType</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Drupal\Core\Field\FieldStorageDefinitionInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Drupal\Core\Field\Plugin\Field\FieldType\EntityReferenceItem</span> <span class="k">as</span> <span class="nc">CoreEntityReferenceItem</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Drupal\Core\TypedData\DataReferenceTargetDefinition</span><span class="p">;</span>
<span class="kn">use</span> <span class="nf">Drupal\my_module</span><span class="err">\</span><span class="nc">Plugin\DataType\CacheableEntityReference</span><span class="p">;</span>
<span class="cd">/**
* Overridden entity reference item class.
*/</span>
<span class="k">final</span> <span class="kd">class</span> <span class="nc">EntityReferenceItem</span> <span class="kd">extends</span> <span class="nc">CoreEntityReferenceItem</span> <span class="p">{</span>
<span class="cd">/**
* {@inheritDoc}
*/</span>
<span class="k">public</span> <span class="k">static</span> <span class="k">function</span> <span class="n">propertyDefinitions</span><span class="p">(</span><span class="kt">FieldStorageDefinitionInterface</span> <span class="nv">$field_definition</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$definitions</span> <span class="o">=</span> <span class="k">parent</span><span class="o">::</span><span class="nf">propertyDefinitions</span><span class="p">(</span><span class="nv">$field_definition</span><span class="p">);</span>
<span class="nv">$definitions</span><span class="p">[</span><span class="s1">'entity'</span><span class="p">]</span><span class="o">-></span><span class="nf">setClass</span><span class="p">(</span><span class="nc">CacheableEntityReference</span><span class="o">::</span><span class="n">class</span><span class="p">);</span> <span class="c1">// This is the key</span>
<span class="k">return</span> <span class="nv">$definitions</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="k">declare</span><span class="p">(</span><span class="n">strict_types</span><span class="o">=</span><span class="mi">1</span><span class="p">);</span>
<span class="kn">namespace</span> <span class="nn">Drupal\my_module\Plugin\DataType</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Drupal\Core\Cache\RefinableCacheableDependencyInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Drupal\Core\Cache\RefinableCacheableDependencyTrait</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Drupal\Core\Entity\Plugin\DataType\EntityReference</span><span class="p">;</span>
<span class="k">final</span> <span class="kd">class</span> <span class="nc">CacheableEntityReference</span> <span class="kd">extends</span> <span class="nc">EntityReference</span> <span class="kd">implements</span> <span class="nc">RefinableCacheableDependencyInterface</span> <span class="p">{</span>
<span class="kn">use</span> <span class="nc">RefinableCacheableDependencyTrait</span><span class="p">;</span> <span class="c1">// This adds default cache handling methods/parameters.</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Create a computed field definition and implement a list class to perform the computation.</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cd">/**
* Implements hook_entity_bundle_field_info().
*/</span>
<span class="k">function</span> <span class="n">my_module_entity_bundle_field_info</span><span class="p">(</span><span class="kt">EntityTypeInterface</span> <span class="nv">$entity_type</span><span class="p">,</span> <span class="nv">$bundle</span><span class="p">,</span> <span class="kt">array</span> <span class="nv">$base_field_definitions</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$entity_type</span><span class="o">-></span><span class="nf">id</span><span class="p">()</span> <span class="o">===</span> <span class="s1">'entity_type'</span> <span class="o">&&</span> <span class="nv">$bundle</span> <span class="o">===</span> <span class="s1">'bundle'</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">[</span>
<span class="s1">'computed_er_field'</span> <span class="o">=></span> <span class="nc">BundleFieldDefinition</span><span class="o">::</span><span class="nf">create</span><span class="p">(</span><span class="s1">'entity_reference'</span><span class="p">)</span>
<span class="o">-></span><span class="nf">setLabel</span><span class="p">(</span><span class="s1">'Related Entity'</span><span class="p">)</span>
<span class="o">-></span><span class="nf">setComputed</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">)</span>
<span class="o">-></span><span class="nf">setCardinality</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="o">-></span><span class="nf">setDescription</span><span class="p">(</span><span class="s1">'This is a pretty cool way to expose data in JSON:API.'</span><span class="p">)</span>
<span class="o">-></span><span class="nf">setSetting</span><span class="p">(</span><span class="s1">'target_type'</span><span class="p">,</span> <span class="s1">'target_entity_type'</span><span class="p">)</span>
<span class="o">-></span><span class="nf">setClass</span><span class="p">(</span><span class="nc">MyModuleComputedFieldItemList</span><span class="o">::</span><span class="n">class</span><span class="p">),</span>
<span class="p">];</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="k">declare</span><span class="p">(</span><span class="n">strict_types</span><span class="o">=</span><span class="mi">1</span><span class="p">);</span>
<span class="kn">namespace</span> <span class="nn">Drupal\my_module</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Drupal\Core\Cache\CacheableMetadata</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Drupal\Core\Field\EntityReferenceFieldItemList</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Drupal\Core\TypedData\ComputedItemListTrait</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Drupal\profile\Entity\ProfileInterface</span><span class="p">;</span>
<span class="k">final</span> <span class="kd">class</span> <span class="nc">MyModuleComputedFieldItemList</span> <span class="kd">extends</span> <span class="nc">EntityReferenceFieldItemList</span> <span class="p">{</span>
<span class="kn">use</span> <span class="nc">ComputedItemListTrait</span><span class="p">;</span>
<span class="cd">/**
* {@inheritDoc}
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">getConstraints</span><span class="p">()</span> <span class="p">{</span>
<span class="c1">// This is purely computed.</span>
<span class="k">return</span> <span class="p">[];</span>
<span class="p">}</span>
<span class="cd">/**
* {@inheritDoc}
*/</span>
<span class="k">protected</span> <span class="k">function</span> <span class="n">computeValue</span><span class="p">()</span> <span class="p">{</span>
<span class="c1">// Dependency injection is unavailable here.</span>
<span class="c1">// @see https://www.drupal.org/project/drupal/issues/2053415</span>
<span class="nv">$entity</span> <span class="o">=</span> <span class="nc">LoadYourEntity</span><span class="o">::</span><span class="nf">here</span><span class="p">();</span>
<span class="nv">$item</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="nf">createItem</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="p">[</span><span class="s1">'target_id'</span> <span class="o">=></span> <span class="nv">$entity</span><span class="p">]);</span>
<span class="nv">$item</span><span class="o">-></span><span class="nf">get</span><span class="p">(</span><span class="s1">'entity'</span><span class="p">)</span><span class="o">-></span><span class="nf">addCacheableDependency</span><span class="p">(</span>
<span class="cm">/* Add whatever logic is necessary, here. */</span>
<span class="p">);</span>
<span class="c1">// @phpstan-ignore-next-line</span>
<span class="nv">$this</span><span class="o">-></span><span class="k">list</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="nv">$item</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="making-it-work-for-entity-references">Making it work for entity references</h2>
<p>That’s pretty cool, but the metadata doesn’t yet serialize properly for entity references.
There is an MR/patch on <a href="https://www.drupal.org/project/drupal/issues/3252278">this issue</a>
which replicates the serialization logic from the issue referenced above, but
for entity references.</p>
<p>Happy reference computation!</p>Brad Jonesbrad@kinksters.datingThe data model for Meet Kinksters’ backend requires us to keep track of many entity relationships, e.g. users who have shown interest in each other. Many relationships are appropriate to maintain in a database entry, because they are integral to the entity itself (matches are a good example.)A trait for always validating entities on save2022-08-13T00:00:00+00:002022-08-13T00:00:00+00:00https://tech.kinksters.dating/posts/trait-always-validate-entity-save-drupal<p>Drupal features a powerful entity (more broadly, typed data) entity validation API, powered
by the Symfony Validation component. Problem is, Drupal hardly ever uses it.</p>
<p>One might assume that entities are valdiated on save, but by-in-large they are not. This
behavior is opt-in, by calling <code class="language-plaintext highlighter-rouge">FieldableEntityInterface::setValidationRequired()</code>,
but core only “sort of” uses it in ContentEntityForm. I say “sort of,” because the validation
is performed in the context of validating the form, and the flag only appears to be set
in the spirit of completeness during that operation.</p>
<p>But why not validate more? I can understand how this would be a major change in Drupal core,
but there’s nothing stopping you from implementing a fail-earlier approach in your own
codebase. In fact, if you’re utilizing JSON:API module, posting new content already validates
the full entity, not just provided fields.</p>
<h2 id="a-trait-to-always-validate-an-entity-on-save">A trait to always validate an entity on save</h2>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="k">declare</span><span class="p">(</span><span class="n">strict_types</span><span class="o">=</span><span class="mi">1</span><span class="p">);</span>
<span class="kn">namespace</span> <span class="nn">Kinksters</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Drupal\Core\Entity\EntityStorageInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\Validator\ConstraintViolation</span><span class="p">;</span>
<span class="cd">/**
* Trait for content entities requiring validation prior to save.
*/</span>
<span class="kd">trait</span> <span class="nc">ContentEntityRequiringValidationTrait</span> <span class="p">{</span>
<span class="cd">/**
* Pre-save method for performing validation.
*
* @param \Drupal\Core\Entity\EntityStorageInterface $storage
* Entity storage.
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">preSave</span><span class="p">(</span><span class="kt">EntityStorageInterface</span> <span class="nv">$storage</span><span class="p">):</span> <span class="kt">void</span> <span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="nf">setValidationRequired</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nv">$this</span><span class="o">-></span><span class="n">validated</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$validation</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="nf">validate</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(</span><span class="nb">count</span><span class="p">(</span><span class="nv">$validation</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$message</span> <span class="o">=</span> <span class="nb">sprintf</span><span class="p">(</span>
<span class="s1">'%s failed validation: %s'</span><span class="p">,</span>
<span class="nv">$this</span><span class="o">-></span><span class="nf">getEntityType</span><span class="p">()</span><span class="o">-></span><span class="nf">getLabel</span><span class="p">(),</span>
<span class="nb">implode</span><span class="p">(</span><span class="s1">', '</span><span class="p">,</span> <span class="nb">array_map</span><span class="p">(</span>
<span class="k">fn</span> <span class="p">(</span><span class="kt">ConstraintViolation</span> <span class="nv">$violation</span><span class="p">)</span> <span class="o">=></span> <span class="nv">$violation</span><span class="o">-></span><span class="nf">getMessage</span><span class="p">(),</span>
<span class="nb">iterator_to_array</span><span class="p">(</span><span class="nv">$validation</span><span class="p">)</span>
<span class="p">))</span>
<span class="p">);</span>
<span class="k">throw</span> <span class="k">new</span> <span class="err">\</span><span class="nf">LogicException</span><span class="p">(</span><span class="nv">$message</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">parent</span><span class="o">::</span><span class="nf">preSave</span><span class="p">(</span><span class="nv">$storage</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Now, simply <code class="language-plaintext highlighter-rouge">use ContentEntityRequiringValidationTrait;</code> in your content entity class,
and the <code class="language-plaintext highlighter-rouge">preSave()</code> method will validate the entity. The parent method will further
ensure the validation actually occurred, or throw its own <code class="language-plaintext highlighter-rouge">LogicException</code>.</p>
<h2 id="wait-what-about-patching-an-invalid-entity-with-jsonapi">Wait, what about <code class="language-plaintext highlighter-rouge">PATCH</code>ing an invalid entity with JSON:API?</h2>
<p>It’s important to be able to still <code class="language-plaintext highlighter-rouge">PATCH</code> fields on an otherwise invalid entity over
JSON:API, otherwise you’d be stuck with no way to remedy the incompleteness.
This trait won’t block this because JSON:API performs validation of the entity
in <a href="https://git.drupalcode.org/project/drupal/-/blob/d6da999e3582949fa09e52ba985b3e1597e51267/core/modules/jsonapi/src/Controller/EntityResource.php#L341"><code class="language-plaintext highlighter-rouge">EntityResource::patchIndividual()</code></a>,
but only throws exceptions if the explicitly patched fields fail. The entity’s
internal <code class="language-plaintext highlighter-rouge">validated</code> property is set, meaning our <code class="language-plaintext highlighter-rouge">::preSave()</code> implementation
above will not perform a second, full entity validation.</p>Brad Jonesbrad@kinksters.datingDrupal features a powerful entity (more broadly, typed data) entity validation API, powered by the Symfony Validation component. Problem is, Drupal hardly ever uses it.A quick-and-easy CloudFront mock for locally testing signed URLs2022-04-22T00:00:00+00:002022-04-22T00:00:00+00:00https://tech.kinksters.dating/posts/mocking-cloudfront-signed-urls<p>Security is a major concern in building a dating app. Part of that is ensuring
sensitive customer data, including photos, are only shared with users with proper
access. (That restriction might just be “registered users,” but that still
requires access checking, e.g., when users block one another.) Meet Kinksters will
also launch with a unique feature allowing your photos to be blurred until you
show interest in or match with someone, further complicating our access regime.</p>
<p>To protect user photos, and improve our performance, we leverage Amazon CloudFront’s
<a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-urls.html">signed URL</a>
feature. This allows us to cache user photos in Amazon POPs closer to clients as well
as limit access based on signed query parameters. But how to test this locally?</p>
<p>CloudFront is, at its core, just a caching reverse proxy, so there’s no real
“official” type of self-hosted version; you’d just use Nginx. That is exactly what
I did when I started to test our photo service locally. However, as I got deeper into
implementing the <a href="http://localhost:8088/posts/2022-04-09-networking-android-react-native/">client side</a>,
I realized that while Nginx was a good caching mock, it lacked an easy way for me to
test whether a signed URL was expired. I’ve generally set my various expiration times
(e.g., bearer token expiration) to abnormally low values in local development - just
a few minutes - to test this functionality more frequently and stress-test code paths
that might only run every few hours or days, otherwise.</p>
<p>It turns out Nginx’s conventional build does not provide a runtime that can compare
a query parameter’s value against the current time. This <em>is</em> possible however with
<a href="https://github.com/openresty/lua-nginx-module#access_by_lua_block">Openresty</a>,
a superset of Nginx with all sorts of Lua-powered scripting enabled by
default. I didn’t really find anything good/current on implementing this, hence this
blog post.</p>
<p>It’s worth mentioning that nothing below actually validates the signature, as
CloudFront would do. Since an <code class="language-plaintext highlighter-rouge">Expires</code> query parameter is an integral part of
the signed URL, however, I figure checking this is more than good enough to
achieve my aim of catching expired signatures. I don’t need to validate the
operation of my signing library overall.</p>
<p>The key is an Nginx configuration that uses a Lua block to do a simple comparison
against the expiration time.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>resolver 127.0.0.11;
proxy_cache_path /tmp/nginxcache levels=1:2 keys_zone=STATIC:10m inactive=24h max_size=1g;
server {
listen 80;
access_log on;
location / {
access_by_lua_block {
local args, err = ngx.req.get_uri_args()
if (tonumber(args.Expires) < os.time()) then
ngx.exit(ngx.HTTP_FORBIDDEN)
end
}
proxy_set_header Host $host;
proxy_set_header x-psk ${PSK};
proxy_cache STATIC;
proxy_cache_valid 200 1d;
add_header X-Cache-Date $upstream_http_date;
set $upstream ${UPSTREAM};
proxy_pass http://$upstream:8080;
}
}
</code></pre></div></div>
<p>You’ll notice this configuration is templated. This is in order to dynamically set
the hostname of the upstream (e.g., a service name on the same <code class="language-plaintext highlighter-rouge">docker-compose</code>
project’s network) and a value for a pre-shared key header, which our
<a href="https://github.com/thephpleague/glide">Glide</a> microservice uses to ensure
requests are allowed only from the reverse proxy.</p>
<p>The official Nginx image
<a href="https://github.com/docker-library/docs/blob/master/nginx/README.md#using-environment-variables-in-nginx-configuration-new-in-119">supports templated configuration</a>
out of the box, but while Openresty <a href="https://github.com/openresty/docker-openresty/issues/38">includes <code class="language-plaintext highlighter-rouge">envsubst</code></a>,
it doesn’t run a script (like <code class="language-plaintext highlighter-rouge">library/nginx</code> does) to automagically transliterate the
variables.</p>
<p>Plus, turns out this little helper script is more-or-less necessary, since in the example above,
<code class="language-plaintext highlighter-rouge">set $upstream ${UPSTREAM}</code> would try and replace both variables, even though the first one
needs to get passed through as-is, as valid Nginx config text. So the script collects the defined
environment variables before running <code class="language-plaintext highlighter-rouge">envsubst</code>.</p>
<p>One little caveat: I had to ever-so-slightly tweak
<a href="https://github.com/nginxinc/docker-nginx/blob/b0e153a1b644ca8b2bd378b14913fff316e07cf2/stable/debian/20-envsubst-on-templates.sh#L25">the Nginx script</a>
though, since it redirects output to handle <code class="language-plaintext highlighter-rouge">3</code> which is undefined. Just remove the redirection
and allow the message to print to <code class="language-plaintext highlighter-rouge">STDOUT</code> and you’ll be fine. Include a copy of this script
in your Docker entrypoint before starting Openresty.</p>
<p>Happy mocking!</p>Brad Jonesbrad@kinksters.datingSecurity is a major concern in building a dating app. Part of that is ensuring sensitive customer data, including photos, are only shared with users with proper access. (That restriction might just be “registered users,” but that still requires access checking, e.g., when users block one another.) Meet Kinksters will also launch with a unique feature allowing your photos to be blurred until you show interest in or match with someone, further complicating our access regime.Using `mitmproxy` for better native-app traffic debugging2022-04-22T00:00:00+00:002022-04-22T00:00:00+00:00https://tech.kinksters.dating/posts/mitmproxy-better-network-interception<p>As I work more on the native app for Meet Kinksters, it’s been important to
debug network traffic between it and various backend services.</p>
<p>Especially when building with Expo, there are a few
<a href="https://docs.expo.dev/workflow/debugging/#inspecting-network-traffic">options</a>
depending on your execution environment. When doing initial development with
Expo Go, the React Native Debugger effectively runs your code inside of
Chrome, and network requests can be traced there. (Beware, if your backend
sets cookies, Chrome will store and use them!)</p>
<p>The Expo docs recommend Flipper for debugging a bare app, and it generally
works well. However, I’ve found the network inspector’s layout to be a bit
clunky and there are <a href="https://github.com/facebook/flipper/issues/348">known bugs</a>
which result in annoying duplicate table entries. Also, if I’m testing on a
physical device not connected via USB, only the React Native logs are available,
anyway.</p>
<p>The Expo docs link to a tutorial on using <code class="language-plaintext highlighter-rouge">mitmproxy</code>, however it suggests
installing a system-wide proxy on your phone which may be overkill. In my case,
I mostly care about viewing traffic to my main app backend. (If I need a more
global view of all my microservices, I can just use Flipper and the emulator.)
This led me to running <code class="language-plaintext highlighter-rouge">mitmproxy</code> in reverse proxy mode, and simply adding
it to my <code class="language-plaintext highlighter-rouge">docker-compose</code> project and pointing the app at that port.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">services</span><span class="pi">:</span>
<span class="na">mitmproxy</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">mitmproxy/mitmproxy</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="na">http_proxy</span><span class="pi">:</span> <span class="s">http://web:8080</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">127.0.0.1:8090:8081"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">${WEB_HOST_BASE}:8083:8080"</span>
<span class="na">command</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">mitmweb</span>
<span class="pi">-</span> <span class="s">--web-host</span>
<span class="pi">-</span> <span class="s">0.0.0.0</span>
<span class="pi">-</span> <span class="s">--mode</span>
<span class="pi">-</span> <span class="s">reverse:http://web:8080</span>
<span class="pi">-</span> <span class="s">--set</span>
<span class="pi">-</span> <span class="s">keep_host_header=true</span>
</code></pre></div></div>
<p>The web UI will be available at <code class="language-plaintext highlighter-rouge">http://localhost:8090</code> in this example.</p>
<p>The <code class="language-plaintext highlighter-rouge">keep_host_header</code> setting is important if your app generates URLs based
on the current request (and makes <code class="language-plaintext highlighter-rouge">mitmproxy</code> act more like a production
reverse proxy in that regard.) I don’t believe it sets headers such as
<code class="language-plaintext highlighter-rouge">x-forwarded-for</code> out of the box, however.</p>
<p>Nothing revolutionary here, but maybe this will give you an additional option
to consider when developing your own app.</p>Brad Jonesbrad@kinksters.datingAs I work more on the native app for Meet Kinksters, it’s been important to debug network traffic between it and various backend services.Networking for real-device React Native testing on-the-go2022-04-09T00:00:00+00:002022-04-09T00:00:00+00:00https://tech.kinksters.dating/posts/networking-android-react-native<p>As I build Meet Kinksters, one personal goal is to share the knowledge I develop along
the way with others. So much of the details of building a complex app, especially with a
native app client, are difficult to ordain without experience. So many of the tutorials
I find online regarding the “tough problems” of decoupled web either paper over this
complexity or offer only simplistic, not-production-ready solutions that omit important
details like state management.</p>
<h2 id="real-device-testing-and-the-local-network">Real-device testing and the local network</h2>
<p>One such sticky wicket has been how to handle networking when testing on a “real” device,
in this case my Android phone. In an earlier blog post, I shared how I
<a href="/posts/2021-11-25-expo-docker-emulator">was able to use the Expo Go app with Dockerized build tools</a>
to get started with my mobile app. This helped get me going, but once I needed to implement
and test <a href="https://docs.expo.dev/versions/latest/sdk/in-app-purchases/">in-app payments</a>, I had to
“graduate” to a <a href="https://docs.expo.dev/development/build/#with-eas">development build</a>. Furthermore,
in-app payments are testable only on physical devices (at least for Android).</p>
<p>When running in an emulator, I was able to bind the Metro bundler and all my app services to ports
on the Docker bridge interface. When I started testing on my physical device, I changed those bindings
to <code class="language-plaintext highlighter-rouge">0.0.0.0</code>, or all interfaces, and added a DNS record to my home (well, RV) router to resolve the
<code class="language-plaintext highlighter-rouge">docker-bridge</code> hostname I’d been using to my laptop’s local network IP.</p>
<p>That’s well and good, but what happens when I’m off my local, trusted WiFi? I travel frequently,
working from hotels and coffee-shops and libraries. Though it’s a relatively low risk, working
similarly on these foreign networks means I’d have to make my configuration much more dynamic,
if my devices could talk to one another to begin with. Then there’s the question of DNS resolution.
There’s no real way to add custom host records in Android without rooting, and solutions like
creating a <a href="https://shadowsocks.org/en/index.html">Shadowsocks VPN</a> from my phone to my laptop
started feeling very janky and over-engineered.</p>
<h2 id="tethering-to-the-rescue">Tethering to the rescue</h2>
<p>After going down a few rabbit-holes (VPNs, custom DNS records/Unbound/Stubby, etc.) I remembered
that my phone can do both Bluetooth and USB tethering, in addition to running a WiFi hotspot. What
if I could make my laptop and phone share a private subnet, while allowing both to stay
connected to the Internet over the hotel’s network? And keeping my VPN enabled, to boot?</p>
<p>The below solution works for both USB and Bluetooth tethering; Bluetooth seems a little slower,
but it lets me skip a cable and doesn’t drop out when I bump the cable.</p>
<p>I’m on Pop! OS (basically, Ubuntu) but this should work on other OSes, too.</p>
<ol>
<li>Enable bluetooth or USB tethering on the device.</li>
<li>Connect the laptop to the device, and edit the routing rules to enable
“Use this connection only for resources on its network.” This will keep all your other traffic
routed through your default route.</li>
<li>Find the IP issued to the laptop from the phone. E.g.:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ ifconfig bnep0
bnep0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.44.226 netmask 255.255.255.0 broadcast 192.168.44.255
</code></pre></div> </div>
</li>
<li>Configure Metro bundler, your app backend, and any other services your app needs over the network
to bind to this IP. I want to avoid binding to <em>all</em> interfaces, including the hotel WiFi, but
this also needs to be dynamic. So, I’ve changed my <code class="language-plaintext highlighter-rouge">docker-compose.yml</code> files to interpolate
an environment variable. If I set it when running <code class="language-plaintext highlighter-rouge">docker-compose up</code>, that value is used.
Otherwise, I have set a default for this environment variable in my <code class="language-plaintext highlighter-rouge">.envrc</code> file to the Docker bridge
IP, as before.
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">services</span><span class="pi">:</span>
<span class="na">web</span><span class="pi">:</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">127.0.0.1:8082:8080"</span> <span class="c1"># So I can always just do localhost:8082 for the web UI.</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">${WEB_HOST_BASE}:8082:8080"</span> <span class="c1"># The IP for the tethered interface, or the Docker Bridge.</span>
</code></pre></div> </div>
</li>
<li>Ensure the app also uses this environment variable in setting self-referential URLs, e.g. for
CDN mocks, external services, etc.</li>
<li>Start Expo and the app backend, setting the base IP to that of the tethered interface:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ WEB_HOST_BASE=192.168.44.226 docker-compose up
</code></pre></div> </div>
</li>
</ol>
<p>This not only allows me to securely test my app without exposing it to the WiFi network, but it enforces
12-Factor App discipline and removes hard-coded hostname strings from my codebase.</p>Brad Jonesbrad@kinksters.datingAs I build Meet Kinksters, one personal goal is to share the knowledge I develop along the way with others. So much of the details of building a complex app, especially with a native app client, are difficult to ordain without experience. So many of the tutorials I find online regarding the “tough problems” of decoupled web either paper over this complexity or offer only simplistic, not-production-ready solutions that omit important details like state management.Programmatically manage bundle fields with `ConfigurableFieldManager`2022-03-29T00:00:00+00:002022-03-29T00:00:00+00:00https://tech.kinksters.dating/posts/programmatically-managing-drupal-bundle-fields<p>Drupal’s ORM and Field API is rather powerful, but it can be a bear to understand
after entities and their fields are initially installed. If you’re writing a custom
entity type, its base fields are pulled from <code class="language-plaintext highlighter-rouge">ContentEntityInterface::baseFieldDefinitions()</code>.
Bundle fields can either be added through the <code class="language-plaintext highlighter-rouge">field_ui</code> module’s interface
or programmatically with <code class="language-plaintext highlighter-rouge">entity</code> module’s bundle plugins.</p>
<p>What if you want to add or edit fields after the fact? Drupal’s documentation
<a href="https://www.drupal.org/docs/drupal-apis/update-api/updating-entities-and-fields-in-drupal-8">covers adding and editing base fields</a>,
but bundle fields are different still.</p>
<p>This blog post exists simply to call your attention to Commerce module’s excellent
<a href="https://git.drupalcode.org/project/commerce/-/blob/04d7c675aa617e590fd0679eaa62e0f00f707ac9/src/ConfigurableFieldManager.php"><code class="language-plaintext highlighter-rouge">ConfigurableFieldManager</code></a>,
which is not at all Commerce-specific. Use it to update bundle fields, either as part of
an update hook or in a one-off if you’re working in custom code that does not require
an upgrade path:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kn">use</span> <span class="nc">Drupal\Core\Field\FieldStorageDefinitionInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Drupal\entity\BundleFieldDefinition</span><span class="p">;</span>
<span class="nv">$field_storage_definition</span> <span class="o">=</span> <span class="nc">BundleFieldDefinition</span><span class="o">::</span><span class="nf">create</span><span class="p">(</span><span class="s1">'commerce_remote_id'</span><span class="p">)</span>
<span class="o">-></span><span class="nf">setName</span><span class="p">(</span><span class="s1">'remote_id'</span><span class="p">)</span>
<span class="o">-></span><span class="nf">setTargetEntityTypeId</span><span class="p">(</span><span class="s1">'entity_type_id'</span><span class="p">)</span>
<span class="o">-></span><span class="nf">setTargetBundle</span><span class="p">(</span><span class="s1">'bundle_id'</span><span class="p">)</span>
<span class="o">-></span><span class="nf">setLabel</span><span class="p">(</span><span class="s1">'Remote ID'</span><span class="p">)</span>
<span class="o">-></span><span class="nf">setCardinality</span><span class="p">(</span><span class="nc">FieldStorageDefinitionInterface</span><span class="o">::</span><span class="no">CARDINALITY_UNLIMITED</span><span class="p">)</span>
<span class="o">-></span><span class="nf">setRequired</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">)</span>
<span class="o">-></span><span class="nf">setDisplayConfigurable</span><span class="p">(</span><span class="s1">'form'</span><span class="p">,</span> <span class="kc">FALSE</span><span class="p">)</span>
<span class="o">-></span><span class="nf">setDisplayConfigurable</span><span class="p">(</span><span class="s1">'view'</span><span class="p">,</span> <span class="kc">FALSE</span><span class="p">);</span>
<span class="cd">/** @var \Drupal\commerce\ConfigurableFieldManagerInterface $configurableFieldManager */</span>
<span class="nv">$configurableFieldManager</span> <span class="o">=</span> <span class="err">\</span><span class="nc">Drupal</span><span class="o">::</span><span class="nf">service</span><span class="p">(</span><span class="s1">'commerce.configurable_field_manager'</span><span class="p">);</span>
<span class="nv">$configurableFieldManager</span><span class="o">-></span><span class="nf">createField</span><span class="p">(</span><span class="nv">$field_storage_definition</span><span class="p">);</span>
</code></pre></div></div>
<p>There’s an issue to <a href="https://www.drupal.org/project/entity/issues/3135737">bring this code into Entity API module</a>,
but there’s no need to wait.</p>
<p>This great little helper service follows in a fine tradition of Commerce-born
enhancements to the Entity API. I’ve found clever uses for the <a href="https://git.drupalcode.org/project/commerce/-/blob/04d7c675aa617e590fd0679eaa62e0f00f707ac9/src/EntityTraitManagerInterface.php">Entity Traits</a>
pattern in projects, for instance. The above example also leverages Commerce Core’s
remote ID field type, applicable in many non-commerce applications.</p>
<p>Work smarter, not harder!</p>
<p>(Funny story about that mindset: I once was being introduced to a company’s staff
after joining as a consultant, and I included “An advocate for working smarter, not harder”
in my slide deck. Management took it out. I don’t work there anymore. They apparently
liked working harder for its own sake.)</p>Brad Jonesbrad@kinksters.datingDrupal’s ORM and Field API is rather powerful, but it can be a bear to understand after entities and their fields are initially installed. If you’re writing a custom entity type, its base fields are pulled from ContentEntityInterface::baseFieldDefinitions(). Bundle fields can either be added through the field_ui module’s interface or programmatically with entity module’s bundle plugins.Invalidating Drupal 8+ cache items by owner2022-03-28T00:00:00+00:002022-03-28T00:00:00+00:00https://tech.kinksters.dating/posts/invalidating-cache-owner-drupal<p>One of the key enhancements in Drupal 8+ over prior versions is the Cache API.
The concept of cache tags and contexts is one of Drupal’s great contributions
to the web ecosystem and is, in my humble opinion, best-in-class.</p>
<p>That said, the out-of-the-box implementation isn’t perfect. Especially if you
implement capital-V views (that is, views from Views module) and/or have frequent
invalidation, some customization of your architecture and implementation are in
order.</p>
<p>As a dating site, Meet Kinksters has caching considerations unlike brochureware
or e-commerce installs. Since the majority of our content is user-generated, we
cannot make assumptions about the frequency or nature of invalidations, but rather
must plan to cache the data that provides us the most significant performance
improvement. Importantly, this means changing our caching paradigm away from
rendered content (this is relatively cheap to generate) but rather data which is
accessed in frequent code paths, such as resolving a user’s access.</p>
<h2 id="invalidating-based-on-the-future-event">Invalidating based on the future event</h2>
<p>Out of the box, Drupal can invalidate objects based on their identity
(e.g., <code class="language-plaintext highlighter-rouge">node:1</code>) or entire classes of objects based on their type (e.g.,
<code class="language-plaintext highlighter-rouge">node_blog_list</code>). This
<a href="https://www.drupal.org/project/drupal/issues/3055371">isn’t without its own performance considerations</a>,
but does cover many default use cases and is easy to understand.
Don’t confuse tags with contexts (this was a favorite interview
question when I was a CTO), the latter of which determines <em>under what conditions
the cached data may vary</em>.</p>
<p>But what of invalidation that must occur based on related entities,
for instance a user’s profiles? And more specifically, what if the
user has yet to create any profiles? We can’t include those related entities
in the referencing entity’s cache tags, because we don’t know their ID(s).
Furthermore, we don’t want to invalidate based on the list cache tag (though
this would “work”), because it would trigger on <em>every</em> profile, not just
those of the user in question.</p>
<p>This is where Drupal’s Cache API begins to shine. <em>Cache tags are just strings of text</em>,
and so we can create custom tags and invalidate against them at will.</p>
<p>Using this important knowledge, we can craft cache tags that invalidate
classes of objects based on their characteristics, and because tags are simply
flags sent to the invalidation system, <em>new</em> entities can invalidate
dependent objects simply by virtue of their creation. This is very much like the
<code class="language-plaintext highlighter-rouge">[entity_type_id]_list</code> “list cache tags” supported out of the box, but
refined to a narrower set.</p>
<h2 id="a-simple-utility-to-create-cache-tags-to-invalidate-per-owner">A simple utility to create cache tags to invalidate per owner.</h2>
<p>In my earlier example, we pull through certain data from the user’s dating
profile when assessing access by user, and conversely user profiles contain
computed sanitized data from a user’s account (e.g., their age, so as not to
directly reveal their birthdate.) We need to invalidate each object when the
other is saved, created or deleted.</p>
<p>A simple utility class allows for the programmatic creation of this custom
cache tag, as well as giving our IDE a method call to index, so we can quickly
search for uses of this pattern in our codebase.</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="k">declare</span><span class="p">(</span><span class="n">strict_types</span><span class="o">=</span><span class="mi">1</span><span class="p">);</span>
<span class="kn">namespace</span> <span class="nn">Drupal\kinksters_system</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Drupal\Core\Entity\ContentEntityInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Drupal\user\EntityOwnerInterface</span><span class="p">;</span>
<span class="cd">/**
* Helper to generate entity owner cache tags to invalidate on save.
*/</span>
<span class="k">final</span> <span class="kd">class</span> <span class="nc">EntityOwnerCacheTags</span> <span class="p">{</span>
<span class="cd">/**
* Generate cache tags to invalidate on save for an entity.
*
* @param \Drupal\Core\Entity\ContentEntityInterface $entity
* Entity.
*
* @return string[]
* Cache tags to invalidate.
*/</span>
<span class="k">public</span> <span class="k">static</span> <span class="k">function</span> <span class="n">getCacheTagsForEntity</span><span class="p">(</span><span class="kt">ContentEntityInterface</span> <span class="nv">$entity</span><span class="p">):</span> <span class="kt">array</span> <span class="p">{</span>
<span class="nb">assert</span><span class="p">(</span><span class="nv">$entity</span> <span class="k">instanceof</span> <span class="nc">EntityOwnerInterface</span><span class="p">);</span>
<span class="k">return</span> <span class="k">self</span><span class="o">::</span><span class="nf">getCacheTags</span><span class="p">(</span>
<span class="nv">$entity</span><span class="o">-></span><span class="nf">getEntityTypeId</span><span class="p">(),</span>
<span class="nv">$entity</span><span class="o">-></span><span class="nf">getOwnerId</span><span class="p">(),</span>
<span class="nv">$entity</span><span class="o">-></span><span class="nf">get</span><span class="p">(</span><span class="nv">$entity</span><span class="o">-></span><span class="nf">getEntityType</span><span class="p">()</span><span class="o">-></span><span class="nf">getKey</span><span class="p">(</span><span class="s1">'bundle'</span><span class="p">))</span><span class="o">-></span><span class="nf">getString</span><span class="p">()</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="cd">/**
* Get cache tags to invalidate given an entity, owner and optional bundle.
*
* @param string $entityTypeId
* Entity type ID.
* @param int $entityOwnerId
* Entity Owner ID.
* @param string|NULL $bundle
* Bundle, if any.
*
* @return string[]
* Cache tags to invalidate.
*/</span>
<span class="k">public</span> <span class="k">static</span> <span class="k">function</span> <span class="n">getCacheTags</span><span class="p">(</span><span class="kt">string</span> <span class="nv">$entityTypeId</span><span class="p">,</span> <span class="kt">int</span> <span class="nv">$entityOwnerId</span><span class="p">,</span> <span class="kt">?string</span> <span class="nv">$bundle</span> <span class="o">=</span> <span class="kc">NULL</span><span class="p">):</span> <span class="kt">array</span> <span class="p">{</span>
<span class="k">return</span> <span class="nv">$bundle</span>
<span class="o">?</span> <span class="p">[</span><span class="nb">sprintf</span><span class="p">(</span><span class="s1">'%s_%s:owner:%s'</span><span class="p">,</span> <span class="nv">$entityTypeId</span><span class="p">,</span> <span class="nv">$bundle</span><span class="p">,</span> <span class="nv">$entityOwnerId</span><span class="p">)]</span>
<span class="o">:</span> <span class="p">[</span><span class="nb">sprintf</span><span class="p">(</span><span class="s1">'%s:owner:%s'</span><span class="p">,</span> <span class="nv">$entityTypeId</span><span class="p">,</span> <span class="nv">$entityOwnerId</span><span class="p">)];</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The first method, <code class="language-plaintext highlighter-rouge">::getCacheTagsForEntity()</code>, yields a result we may merge into
an object’s implementation of <code class="language-plaintext highlighter-rouge">EntityInterface::getCacheTagsToInvalidate()</code>.</p>
<p>The second method, <code class="language-plaintext highlighter-rouge">::getCacheTags()</code>, is for use in any context where another
entity type’s cache tags must be invalidated. This could be in a referenced
entity’s <code class="language-plaintext highlighter-rouge">::getCacheTagsForEntity()</code> method, or in custom code that knows it must
invalidate data based on business rules.</p>
<p>It’s a good idea to namespace your cache tags in such a way that they won’t likely
be duplicated in core or contrib patterns, but this is easy enough to avoid.</p>
<p>Happy caching, and may your hit rates always increase.</p>Brad Jonesbrad@kinksters.datingOne of the key enhancements in Drupal 8+ over prior versions is the Cache API. The concept of cache tags and contexts is one of Drupal’s great contributions to the web ecosystem and is, in my humble opinion, best-in-class.json:api, json-rpc and API-first patterns for Drupal2022-02-20T00:00:00+00:002022-02-20T00:00:00+00:00https://tech.kinksters.dating/posts/api-first-patterns-drupal<p>Meet Kinksters is built on top of a wide range of <a href="/about/open-source">open source technologies</a>,
Drupal being chief among them.</p>
<p>This blog post is an attempt at covering the current
state-of-the-art and best practice for “doing API-first” with Drupal, with a json:api focus.</p>
<p>I say “json:api focus” because there is <a href="https://www.drupal.org/project/graphql">solid support for GraphQL</a>
in Drupal, and I’m not here to discount it. That said, Drupal core
<a href="https://www.drupal.org/blog/headless-cms-rest-vs-jsonapi-vs-graphql#5">explicitly prioritizes json:api support over GraphQL</a>
and as you’ll see here, the growing contributed module ecosystem has followed suit.</p>
<h2 id="understanding-the-value-proposition-of-jsonapi">Understanding the value proposition of json:api</h2>
<p>When I first started building decoupled Drupal sites, my instinct was to assume that the
API specification (that is, the json:api spec) covered all aspects of my API, from request
and response shapes to schema to URL structure. In truth, the json:api spec speaks only to
a subset of these concerns. I found clarity in this statement on the project front page:</p>
<blockquote>
<p>If you’ve ever argued with your team about the way your JSON responses should be formatted,
JSON:API can be your anti-<a href="http://bikeshed.org/">bikeshedding</a> tool.</p>
<p>By following shared conventions, you can increase productivity, take advantage of
generalized tooling, and focus on what matters: your application.</p>
<p>…</p>
<p>JSON:API covers creating and updating resources as well, not just responses.</p>
</blockquote>
<p>Bikeshedding is a fun topic worth exploring if you haven’t run into it yet, but the
point here is that json:api is really only concerned with the structure of your payloads.
Not the code that generates them, nor the ORM managing the data, nor the URLs from which
you receive or to which you post those payloads.</p>
<p>Once I understood this important point, the concepts discussed below came into sharper view.</p>
<h2 id="best-practices-for-jsonapi-and-json-rpc-in-drupal">Best practices for json:api and json-rpc in Drupal</h2>
<p>Drupal site owners may enable a feature-rich json:api server by simply enabling the
<code class="language-plaintext highlighter-rouge">jsonapi</code> core module. This simple act might even be sufficient to integrate with
data consumers such as kiosks, aggregators or headless front-ends which access
unauthenticated resources. This is a huge win for very little effort.</p>
<p>As applications grow more complex and require features such as
<a href="https://drupal.org/project/simple_oauth">authentication</a>, write operations and
procedure calls, so too must you adapt your site architecture.</p>
<h3 id="the-temptation-of-custom-controllers">The temptation of custom controllers</h3>
<p>A common first tutorial for novice Drupal developers is to create a
<a href="https://www.drupal.org/docs/creating-custom-modules/step-by-step-tutorial-hello-world/adding-a-basic-controller">custom controller</a>;
this is a quick route to “Hello World” satisfaction. It is unsurprising then that
many developers choose to implement APIs using custom controllers, especially as
Drupal 8+ makes it easy to return JSON response objects instead of HTML.</p>
<p>This pattern quickly grows out of control, and in my experience also encourages
a dangerous blurring of lines between controller logic and Drupal’s access control
APIs. When every “endpoint” is written effectively from scratch, the result is
an unwieldy hodgepodge of unmaintainable code. I can say this with confidence because
I did it for years!</p>
<p>Additionally, what is the shape of the request and response payloads handled by these
custom controllers? By definition, the answer is “whatever you want.” Considering
how easy it is to shortcut Drupal’s built-in APIs for entity CRUD and access, this
approach invites all manner of information disclosure, data corruption and access
bypass.</p>
<p>Let’s return then to json:api’s “anti-bikeshedding” value proposition. Drupal’s
implementation comes with a spec-compliant server with sensible defaults for URL
structure; use it. (And extend as needed.)</p>
<h3 id="when-the-defaults-dont-fit">When the defaults don’t fit</h3>
<p>Meet Kinksters is a dating app, and as such we have a data model which maps rather
well to json:api resources. Things like dating profiles, user blocks, and even payment
methods all have automaticaly-generated json:api resources. But what about a requirement
like our “introductions stack”?</p>
<h4 id="custom-resources">Custom Resources</h4>
<p>Meet Kinksters users may send “introductions,” which are basically unilateral matches.
Users on the free tier may only see one received introduction at a time in their stack,
while paid users can see all their suitors at a glance. This kind of advanced sorting
and access control is beyond Drupal’s out of the box resource collections. But not by
much.</p>
<p>We still want to leverage json:api’s request and response structure, because the returned
data should be parsed according to the spec. In this case, we implement a custom
resource using the <a href="https://www.drupal.org/project/jsonapi_resources">JSON:API Resources module</a>.
Precisely because json:api defines no restriction on the URLs at which your API operates,
custom resources can return any manner of primary and included data based on your
site’s business rules.</p>
<p>Thus, our “introductions stack” custom resource returns documents (Drupal entities)
in the context of the active user and their paid membership status.</p>
<p>Note that json:api doesn’t require that your request even contain a body at all!
Consider a custom endpoint at <code class="language-plaintext highlighter-rouge">/jsonapi/user/{uuid}/block</code> which might perform some
sort of operation to block the other user. (So sad!) The response could contain a
json:api document (say, a block resource, if such a thing exists) or merely
<a href="https://jsonapi.org/format/#document-top-level">reply with a <code class="language-plaintext highlighter-rouge">meta</code> member</a>
with some status information. The specific details are up to you.</p>
<p>Notably, <a href="https://www.drupal.org/project/jsonapi_search_api">json:api Search API</a>
exposes search indexes using custom resources!</p>
<h4 id="compound-responses-and-sparse-fieldsets">Compound responses and sparse fieldsets</h4>
<p>Along a similar vein as custom resources, advanced use cases might prompt your server
to <a href="https://jsonapi.org/format/#document-compound-documents">include related</a>
(but unsolicited) json:api documents in responses. In our case, this
means sending user profile and photo data alongside matches and introductions, because
we know the client will require this information to fully render data in the UI. (What
use is a dating profile without a photo?) Such functionality can be accomplished by
setting default <code class="language-plaintext highlighter-rouge">include</code> parameters in the <a href="https://git.drupalcode.org/project/jsonapi_extras/-/tree/8.x-3.x/modules/jsonapi_defaults">jsonapi_defaults</a>
module.</p>
<p>In this case, the json:api spec is rather permissive, but <a href="https://jsonapi.org/format/#fetching-includes">only if</a>
the client did not send an <code class="language-plaintext highlighter-rouge">include</code> query parameter in the request.
Similar considerations apply to server-initiated <a href="https://jsonapi.org/format/#fetching-sparse-fieldsets">sparse fieldsets</a>.</p>
<h2 id="json-rpc-for-non-resource-operations">JSON-RPC for non-resource operations</h2>
<p>For operations that do not directly map to a Drupal entity, consider json-rpc. Using
the <a href="https://www.drupal.org/project/jsonrpc">contributed module</a>, new methods are
exceedingly easy to create, and similarly easy to implement
<a href="https://www.drupal.org/project/jsonrpc">on the client side</a>.</p>
<h3 id="the-role-of-json-rpc">The role of json-rpc</h3>
<p>Just like my “a-ha” revelation about the value proposition for json:api, so too I needed
experience and comparison to understand the value of json-rpc. Underlying the value of
json-rpc is, well, <a href="https://en.wikipedia.org/wiki/Remote_procedure_call">RPC</a>. There are
many remote procedure call implementations in the wild,
including Google’s <a href="https://grpc.io/about/">gRPC</a>. A dive into the RPC pool will quickly
lead you to ask, do I need HTTP/2? Protobuf? For the web architect needing to choose a
technology stack, this complexity can quickly become overwhelming.</p>
<p>For me, I only ever learned that I needed an RPC protocol because json:api’s <em>resource</em>
focused model no longer matched my needs. In other words, json:api is great at modeling
an API for a <a href="https://en.wikipedia.org/wiki/Object-role_modeling">universe of discourse</a>,
or my application’s entities which might be stored in a relational database.</p>
<p>Actions such as “log in,” “log out” or “clear cache” are examples of procedure calls
which may be undertaken over an API but do not map to
<a href="https://en.wikipedia.org/wiki/Create,_read,_update_and_delete">CRUD</a>
entity operations.</p>
<p>The json-rpc specification is appealing due to its reduced complexity as compared to gRPC.
Method names, parameter values and responses are all serialized as JSON, and lightweight
clients are available in common languages such as PHP and JavaScript. No intermediate
compilation is required.</p>
<h2 id="conclusion-stick-to-the-spec">Conclusion: Stick to the spec</h2>
<p>When building an application that does not provide a public API (that is, you are not
providing your API as-a-service) it can be easy to sidestep spec compliance for
convenience. I believe this is a mistake.</p>
<p>Specifications such as json:api and json-rpc grow out of long-term, consensus-based
work by those with a global view. By using the spec (and sticking to it!) you benefit
from this hard work and consideration of trade-offs you may never have thought of
or not yet encountered. In addition, you ensure your API may be consumed by spec-compliant
clients, instead of tripping over your customizations.</p>
<p>Sticking to the spec might seem like “extra work” (don’t get me started on how this
plays out at penny-pinching agencies!) but I assure you it’s worth doing. If you
find yourself building an implementation that is antipattern to the spec, ask why.
In my personal experience, this introspection has always led me to a better, more
graceful and maintainable solution (and often with far less code.)</p>Brad Jonesbrad@kinksters.datingMeet Kinksters is built on top of a wide range of open source technologies, Drupal being chief among them.