add Namer for DNS SRV records #1611

ccmtaylor · 2017-08-25T14:05:07Z

This adds a plugin that allows linkerd to use SRV records for service discovery. A linkerd.yaml file using the plugin might contain something like this:

routers:
- protocol: http
  dtab: |
    /dnssrv => /#/io.l5d.dnssrv;
    /svc/myservice =>
               /dnssrv/myservice.srv.example.org &
               /dnssrv/myservice2.srv.example.org;
    /svc/other => 
               /dnssrv/other.srv.example.org;

namers:
- kind: io.l5d.dnssrv
  experimental: true

Fixes #1610

hawkw · 2017-08-25T15:44:16Z

Hi @ccmtaylor, this is great, thank you for your contribution!

I'll find some reviewers for this PR; in the mean time, would you mind making sure you've signed our Contributor License Agreement so we can merge this?

hawkw

This basically looks good to me, here are a handful of minor changes you might want to consider. I'd like @olix0r to sign off on this PR as well before we can merge this.

hawkw · 2017-08-25T16:05:05Z

namer/dnssrv/src/main/scala/io/buoyant/namer/dnssrv/DnsSrvNamer.scala

+          NameTree.Leaf(Name.Bound(Var.value(Addr.Bound(srvRecords: _*)), id))
+        }
+      case code =>
+        log.trace("unexpected RCODE: %d for %s", code, address)


I think anything that makes NameTree.Fail probably ought to be logged at the warning level.

hawkw · 2017-08-25T16:12:36Z

namer/dnssrv/src/test/scala/io/buoyant/namer/dnssrv/DnsSrvNamerTest.scala

+    assert(curator.dnsHosts === Some(Seq("localhost")))
+  }
+
+  test("can resolve some public SRV revord") {


I'd feel somewhat more comfortable if tests that create external network connections went in integration rather than unit tests.

hawkw · 2017-08-25T16:16:28Z

namer/dnssrv/src/main/scala/io/buoyant/namer/dnssrv/DnsSrvNamer.scala

+  extends Namer {
+
+  private val log = Logger.get("dnssrv")
+  private val cache = TrieMap.empty[Path, Var[State[NameTree[Name]]]]


Take it or leave it: consider com.twitter.util.Memoize as an alternative to map.getOrElseUpdate.

hawkw · 2017-08-25T16:20:04Z

namer/dnssrv/src/main/scala/io/buoyant/namer/dnssrv/DnsSrvNamerInitializer.scala

+      case Some(hosts) => new DNS.ExtendedResolver(hosts.toArray)
+      case None => new DNS.ExtendedResolver()
+    }
+    resolver.setEDNS(0, 2048, 0, Collections.EMPTY_LIST)


nit: maybe put the level, payload size, etc, in a constant?

ccmtaylor · 2017-08-28T12:26:43Z

namer/dnssrv/src/main/scala/io/buoyant/namer/dnssrv/DnsSrvNamer.scala

    case _ => Activity.value(NameTree.Neg)
  }

-  private[dnssrv] def lookupSrv(address: String, id: Path, residual: Path): Try[NameTree[Name]] = Try {


I've seen other namers hand around a residual path, but I'm not sure how to use it, or why they would.

ccmtaylor · 2017-08-28T12:27:12Z

@hawkw thaks for the quick review! I've addressed your comments; let me know if things look alright.

would you mind making sure you've signed our Contributor License Agreement so we can merge this?

SoundCloud recently signed a CLA. I used my SC email in the commits, but if you prefer, I can close this PR and re-submit from a fork under github.com/soundcloud.

hawkw · 2017-08-28T15:10:39Z

@ccmtaylor:

SoundCloud recently signed a CLA. I used my SC email in the commits, but if you prefer, I can close this PR and re-submit from a fork under github.com/soundcloud.

Oh, okay; I wasn't aware of that and "please sign the CLA" is just in my boilerplate response for all first-time contributors. You're fine, then!

adleong

This is really awesome! Very nice work! I've left a few minor comments below.

adleong · 2017-08-28T21:48:06Z

namer/dnssrv/src/main/scala/io/buoyant/namer/dnssrv/DnsSrvNamer.scala

+
+  private val log = Logger.get("dnssrv")
+  private val memoizedLookup: (Path) => Activity[NameTree[Name]] = Memoize {
+    case path@Path.Utf8(address, _) =>


It looks like you're expecting the path to have exactly 2 segments here. I think this should call .take(1) before the match and then match that one segment.

ok. I actually wrote it that way first, but thought it looked a bit nicer like this because it's one fewer level of indentation (though I guess Utf8.unapply() might need to do more work.

adleong · 2017-08-28T21:48:52Z

namer/dnssrv/src/main/scala/io/buoyant/namer/dnssrv/DnsSrvNamer.scala

+      val id = path.take(1)
+      Activity(Var.async[State[NameTree[Name]]](Activity.Pending) { state =>
+        timer.schedule(refreshInterval) {
+          val next = lookupSrv(address, prefix ++ id) match {


I think you should also pass the residual into this method path.drop(1). (see related comment below)

adleong · 2017-08-28T21:55:13Z

namer/dnssrv/src/main/scala/io/buoyant/namer/dnssrv/DnsSrvNamer.scala

+    case _ => Activity.value(NameTree.Neg)
+  }
+
+  private[dnssrv] def lookupSrv(address: String, id: Path): Try[NameTree[Name]] = Try {


Which part of this method is expected to throw exceptions? It would be nice to scope the Try block more narrowly if possible

resolver.send(query) throws IOException. I'll scope the Try{} to that call.

adleong · 2017-08-28T21:55:35Z

namer/dnssrv/src/main/scala/io/buoyant/namer/dnssrv/DnsSrvNamer.scala

+    case _ => Activity.value(NameTree.Neg)
+  }
+
+  private[dnssrv] def lookupSrv(address: String, id: Path): Try[NameTree[Name]] = Try {


I think this method should take the path residual as a parameter as well.

adleong · 2017-08-28T21:56:10Z

namer/dnssrv/src/main/scala/io/buoyant/namer/dnssrv/DnsSrvNamer.scala

+    val query = DNS.Message.newQuery(question)
+    log.debug("looking up %s", address)
+    val m = resolver.send(query)
+    log.debug("got response %s", address)


I don't think this log is correct

adleong · 2017-08-28T22:09:22Z

namer/dnssrv/src/main/scala/io/buoyant/namer/dnssrv/DnsSrvNamer.scala

+          // valid DNS entry, but no instances.
+          // for some reason, NameTree.Empty doesn't work right
+          log.trace("empty response for %s", address)
+          NameTree.Neg


It's debatable if you want NameTree.Neg here or a leaf with Addr.Bound containing an empty set. The former will consider the name invalid and will fall back to any alternatives in the NameTree. The later will be considered a valid name but any requests will fail because there are no addresses to send to.

a major linkerd use-case for us is to load-balance across two or more deployments of a service (with different SRV records), i.e. the dtab looks like this:

/svc/myservice => /dnssrv/myservice1.srv.example.org & /dnssrv/myservice2.srv.example.org;

if myservice1 resolves to Addr.Bound(Set.empty), would linkerd still call myservice2?

If you use an empty address set then half of the time you'll pick the first branch of the union and fail the request and half of the time you'll pick the second branch of the union and use myservice2

If you use NameTree.Neg then you'll skip to the myservice2 every time.

sounds like NameTree.Neg is the right choice then :). In general, why would linkerd not fallback/load-balance for empty sets? I see this has come up in #1612 and #1549, too.

adleong · 2017-08-28T22:10:41Z

namer/dnssrv/src/main/scala/io/buoyant/namer/dnssrv/DnsSrvNamer.scala

+          NameTree.Neg
+        } else {
+          log.trace("got %d results for %s", srvRecords.length, address)
+          NameTree.Leaf(Name.Bound(Var.value(Addr.Bound(srvRecords: _*)), id))


you want to set the residual as the path of the Name.Bound here

adleong · 2017-08-28T22:11:59Z

namer/dnssrv/src/main/scala/io/buoyant/namer/dnssrv/DnsSrvNamer.scala

+        }
+      case code =>
+        log.warning("unexpected RCODE: %s for %s", DNS.Rcode.string(code), address)
+        NameTree.Fail


I think this should be a Throw(...) instead of a Return(NameTree.Fail)

ok. I'm a little unclear on the behaviour of the different NameTree.* variants. When would I use NameTree.Fail?

It can be useful if you want to artificially halt evaluation of a name tree, but it's rarely used in practice.

adleong · 2017-08-28T22:12:20Z

namer/dnssrv/src/main/scala/io/buoyant/namer/dnssrv/DnsSrvNamerInitializer.scala

+
+  @JsonIgnore
+  override def newNamer(params: Params): Namer = {
+    import org.xbill.DNS


move imports to top of file, please

adleong · 2017-08-28T22:13:16Z

namer/dnssrv/src/integration/scala/io/buoyant/namer/dnssrv/DnsSrvNamerIntegrationTest.scala

+class DnsSrvNamerIntegrationTest extends FunSuite with Matchers {
+  test("can resolve some public SRV revord") {
+    val namer = new DnsSrvNamer(Path.empty, new DNS.ExtendedResolver, new NullTimer, Duration.Zero, new NullStatsReceiver)
+    val result = namer.lookupSrv("_http._tcp.mxtoolbox.com.", Path.read("/foo"))


I think this would be a better black-box test if you use namer.lookup instead.

I agree, but I couldn't figure out how to do something like Future.await() for an Activity. The closest I could find is Activity.sample, but I'm not sure if that would race if the lookup is too slow. I'll push something, but please take a look if that's correct.

if you import io.buoyant.namer.RichActivity then you can do await(namer.lookup(...).toFuture)

not sure if this makes a difference in practice, since Try's `flatMap` also traps exceptions.

ccmtaylor · 2017-08-29T09:38:34Z

@hawkw @adleong: what kind of instrumentation would you expect on namers? I'm passing in a StatsReceiver, but not using it atm. I couldn't see anything consistent across the other namers, but I'm not sure if there exist some metrics across all kinds of namers (e.g. lookup latency, success/failure counts)?

ccmtaylor · 2017-08-29T09:46:41Z

namer/dnssrv/src/integration/scala/io/buoyant/namer/dnssrv/DnsSrvNamerIntegrationTest.scala

+class DnsSrvNamerIntegrationTest extends FunSuite with Awaits with Matchers {
+  test("can resolve some public SRV revord") {
+    val namer = new DnsSrvNamer(Path.empty, new DNS.ExtendedResolver, new NullTimer, Duration.Zero, new NullStatsReceiver)
+    Activity.sample(namer.lookup(Path.read("/_http._tcp.mxtoolbox.com."))) match {


as mentioned in a previous comment, I'd appreciate a +1 that this should work and doesn't just race on the Activity being in Pending state.

ccmtaylor · 2017-08-29T09:51:34Z

namer/dnssrv/src/test/scala/io/buoyant/namer/dnssrv/DnsSrvNamerTest.scala

+  test("parse config") {
+    val yaml = s"""
+                  |kind: io.l5d.dnssrv
+                  |experimental: true


should I make this required? If so, where?

Set experimentalRequired = true in DnsSrvNamerInitializer. See (for example) K8sNamerInitializer:

linkerd/namer/k8s/src/main/scala/io/buoyant/namer/k8s/K8sExternalInitializer.scala

Lines 34 to 35 in 5d22b56

@JsonIgnore

override val experimentalRequired = true

hawkw · 2017-08-29T18:42:00Z

@ccmtaylor we don't currently collect any metrics from namers, although we probably should.

adleong

⭐ This is awesome! Very nice work.

olix0r

massive thanks for this submission.

One small question...

olix0r · 2017-09-01T16:10:51Z

namer/dnssrv/src/main/scala/io/buoyant/namer/dnssrv/DnsSrvNamer.scala

+    )
+    val query = DNS.Message.newQuery(question)
+    log.debug("looking up %s", address)
+    Try(resolver.send(query)) flatMap { message =>


I imagine that resolver.send is a potentially blocking call? If so, then I think DnsSrvNamer should probably be constructed with a com.twitter.util.FuturePool so that these calls don't block the timer thread.

Furthermore, it may be a good idea to record a stat around the lookup times.

olix0r · 2017-09-01T16:11:48Z

namer/dnssrv/src/main/scala/io/buoyant/namer/dnssrv/DnsSrvNamer.scala

+              case Throw(e) => Activity.Failed(e)
+            }
+            state.update(next)
+          }


just to confirm i understand this... timer.schedule returns a closable and the timer is canceled when the Var observation is released?

yes, that is correct

ccmtaylor · 2017-09-04T15:39:25Z

lifting this to a top-level comment, so it doesn't get lost in revisions:

@olix0r: I imagine that resolver.send is a potentially blocking call? If so, then I think DnsSrvNamer should probably be constructed with a com.twitter.util.FuturePool so that these calls don't block the timer thread.

very good point, thanks @olix0r! I tried (and failed) to do this today. It's straight forward to make lookupSrv async with a FuturePool by doing

futurePool(resolver.send(query)) flatMap { message =>
  //...
  Future.value(NameTree.whatever)
  //...
  Future.exception(e)
}

but I can't figure out how to fit Futures into the Activity/Var/Timer APIs :(. Here's what I tried:

I found Activity.future(Future[T]), but that seems to run a future once, and then stop.

I tried to use val (act, witness) = Activity[NameTree[Name]() like this:

// with lookupSrv modified to return Future[Nametree[Name]] as outlined above
case id@Path.Utf8(address) =>
  val (act, witness) = Activity[NameTree[Name]]()
  def loop(): Future[NameTree[Name]] = lookupSrv(address, prefix ++ id, path.drop(1)).respond { result =>
    witness.notify(result)
    timer.doLater(refreshInterval)(loop())
  }
  act

but I'm left with no place to start and evaluate the loop -- calling Await.result(loop()) would miss the point :)

I'm pretty fluent in using Futures, but the Activity and Var APIs are unfamiliar to me, so I'd appreciate any hints.

olix0r · 2017-09-04T16:25:11Z

@ccmtaylor The Var/Activity apis are a bit nuanced and are difficult at times for all of us ;) If you poke around some other namers you may find an idiom like the following:

implicit private[this] val timer = DefaultTimer.twitter

// ...
  Activity(Var.async[Activity.State[NameTree[Name]]](Activity.Pending) { state =>

     @volatile var stopped: Boolean = false

      def loop(): Future[Unit] = {
        if (stopped) Future.Unit
        else pool(resolve(name)).transform { result =>
            result match {
              Ok(addrs) =>
                state() = Activity.Ok(NameTree.Leaf(Addr.Bound(addrs)))

              Throw(e) =>
                log.error(e, "resolution error: %s", name)
                // it may be appropriate to fail the resolution (i.e. Activity.Failed) if there was a pending state before
            }

            // Wait a TTL before resolving again.
            Future.sleep(ttl).before(loop())
        }
      }

      val pending = loop()
      Closable.make { _ =>
        stopped = true

        // Cancel the timer notification or pending requests
        pending.raise(...)

        Future.Unit
      }
  })

... or something roughly like that.

ccmtaylor · 2017-09-04T21:29:40Z

@olix0r thanks for the feedback, that worked great! I found Future.whileDo, which is a nice wrapper around the loop, and I used an AtomicBoolean as shown in the twitter-util cookbook.

use a FuturePool to avoid blocking the timer thread

this should help mitigate effects of transient DNS failures.

olix0r

thanks for putting this together! looks really good

olix0r · 2017-09-05T20:53:33Z

namer/dnssrv/src/main/scala/io/buoyant/namer/dnssrv/DnsSrvNamer.scala

+import com.twitter.util._
+import org.xbill.DNS
+
+class DnsSrvNamer(prefix: Path, resolver: DNS.Resolver, refreshInterval: Duration, stats: StatsReceiver, pool: FuturePool)(implicit val timer: Timer)


nit: prefer to break this very long line into a line per argument

olix0r · 2017-09-05T20:54:45Z

namer/dnssrv/src/main/scala/io/buoyant/namer/dnssrv/DnsSrvNamer.scala

+    val query = DNS.Message.newQuery(question)
+    log.debug("looking up %s", address)
+    pool {
+      val message = Stat.time(latency, TimeUnit.SECONDS)(resolver.send(query))


probably better to measure in milliseconds?

olix0r · 2017-09-05T20:55:21Z

namer/dnssrv/src/main/scala/io/buoyant/namer/dnssrv/DnsSrvNamer.scala

+  private val success = stats.counter("lookup_successes_total")
+  private val failure = stats.counter("lookup_failures_total")
+  private val zeroResults = stats.counter("lookup_zero_results_total")
+  private val latency = stats.stat("request_duration_seconds")


as mentioned below, this should probably be measured in millis (and have suffix _ms like other finagle metrics)

hawkw · 2017-09-06T16:20:49Z

Hi @ccmtaylor, thanks for those last commits yesterday! I'm going to go ahead and merge this, and it'll show up in Linkerd 1.2.0 (which we're expecting to release by September 14th, if not before). Thanks again for your contribution!

ccmtaylor · 2017-09-06T18:29:28Z

Thanks for your time and feedback, @hawkw, @adleong and @olix0r! I just noticed that I made an editing mistake while writing the docs. #1623 fixes it, sorry!

hawkw · 2017-09-06T18:36:31Z

Thanks for the fix, @ccmtaylor!

While writing the docs for #1611, a line was accidentally removed from the Istio docs. This adds it back.

## 1.2.0 2017-09-07 * **Breaking Change**: `io.l5d.mesh`, `io.l5d.thriftNameInterpreter`, linkerd admin, and namerd admin now serve on 127.0.0.1 by default (instead of 0.0.0.0). * **Breaking Change**: Removed support for PKCS#1-formatted keys. PKCS#1 formatted keys must be converted to PKCS#8 format. * Added experimental `io.l5d.dnssrv` namer for DNS SRV records (#1611) * Kubernetes * Added an experimental `io.l5d.k8s.configMap` interpreter for reading dtabs from a Kubernetes ConfigMap (#1603). This interpreter will respond to changes in the ConfigMap, allowing for dynamic dtab updates without the need to run Namerd. * Made ingress controller's ingress class annotation configurable (#1584). * Fixed an issue where Linkerd would continue routing traffic to endpoints of a service after that service was removed (#1622). * Major refactoring and performance improvements to `io.l5d.k8s` and `io.l5d.k8s.ns` namers (#1603). * Ingress controller now checks all available ingress resources before using a default backend (#1607). * Ingress controller now correctly routes requests with host headers that contain ports (#1607). * HTTP/2 * Fixed an issue where long-running H2 streams would eventually hang (#1598). * Fixed a memory leak on long-running H2 streams (#1598) * Added a user-friendly error message when a HTTP/2 router receives a HTTP/1 request (#1618) * HTTP/1 * Removed spurious `ReaderDiscarded` exception logged on HTTP/1 retries (#1609) * Consul * Added support for querying Consul by specific service health states (#1601) * Consul namers and Dtab store now fall back to a last known good state on Consul observation errors (#1597) * Improved log messages for Consul observation errors (#1597) * TLS * Removed support for PKCS#1 keys (#1590) * Added validation to prevent incompatible `disableValidation: true` and `clientAuth` settings in TLS client configurations (#1621) * Changed `io.l5d.mesh`, `io.l5d.thriftNameInterpreter`, linkerd admin, and namerd admin to serve on 127.0.0.1 by default (instead of 0.0.0.0) (#1366) * Deprecated `io.l5d.statsd` telemeter.

When scrollbars are set to always be visible in a browser, we see them appear in the sidebar component of the dashboard. This PR adds CSS that hides the scrollbar for WebKit browsers, i.e., Chrome and Safari and uses an overflow: hidden technique inspired by this solution to hide the scrollbar in Firefox. fixes linkerd#1611 Signed-off-by: Dennis Adjei-Baah <[email protected]>

add Namer for DNS SRV records

956f496

ccmtaylor force-pushed the dnssrv-namer branch from 3578e5a to 956f496 Compare August 25, 2017 14:57

hawkw added this to the 1.2.0 milestone Aug 25, 2017

hawkw requested review from hawkw and olix0r August 25, 2017 16:00

hawkw suggested changes Aug 25, 2017

View reviewed changes

ccmtaylor added 3 commits August 28, 2017 09:33

warning-level logging for unexpected response code

6f35d1d

fixup: take/drop consistent number of path elems

a177b48

extract constants for EDNS configuration

d19305d

ccmtaylor commented Aug 28, 2017

View reviewed changes

ccmtaylor added 2 commits August 28, 2017 16:00

extract integration test

a701320

use Memoize instead of TrieMap to cache lookup activities

bd0c321

ccmtaylor force-pushed the dnssrv-namer branch from 1278eb0 to bd0c321 Compare August 28, 2017 14:00

adleong reviewed Aug 28, 2017

View reviewed changes

ccmtaylor added 3 commits August 29, 2017 09:13

pass residual path to NameTree.Leaf

b2ce90f

reduce scope of Try{} wrapper

258d50e

not sure if this makes a difference in practice, since Try's `flatMap` also traps exceptions.

move imports to top of file

779cef9

ccmtaylor commented Aug 29, 2017

View reviewed changes

ccmtaylor added 2 commits August 31, 2017 16:27

blackbox-test DnsSrvNamer#lookup

c92d2d5

make dnssrv namer experimental

5f0a328

ccmtaylor force-pushed the dnssrv-namer branch from 5a34d87 to 5f0a328 Compare August 31, 2017 14:27

remove bad log line, clarify empty-result return value

39463d1

adleong approved these changes Aug 31, 2017

View reviewed changes

hawkw approved these changes Aug 31, 2017

View reviewed changes

olix0r reviewed Sep 1, 2017

View reviewed changes

ccmtaylor force-pushed the dnssrv-namer branch from 3616376 to a65dec5 Compare September 4, 2017 21:11

ccmtaylor added 2 commits September 4, 2017 23:45

add metrics

960121f

perform DNS lookups asynchronously

eb723b0

use a FuturePool to avoid blocking the timer thread

ccmtaylor force-pushed the dnssrv-namer branch from a65dec5 to eb723b0 Compare September 4, 2017 21:49

ignore failed lookups if previously successful

e129c2c

this should help mitigate effects of transient DNS failures.

olix0r approved these changes Sep 5, 2017

View reviewed changes

ccmtaylor added 3 commits September 6, 2017 09:53

break long constructor into lines

f5dea7d

measure latency in milliseconds

bc16294

add documentation for DNS-SRV namer

d39fe42

hawkw merged commit 3d31437 into linkerd:master Sep 6, 2017

ccmtaylor mentioned this pull request Sep 6, 2017

restore line in Istio docs #1623

Merged

hawkw pushed a commit that referenced this pull request Sep 6, 2017

Re-add line in Istio docs (#1623)

10d36a2

While writing the docs for #1611, a line was accidentally removed from the Istio docs. This adds it back.

ccmtaylor deleted the dnssrv-namer branch September 6, 2017 18:55

hawkw mentioned this pull request Sep 7, 2017

1.2.0 #1629

Merged

ccmtaylor mentioned this pull request Sep 14, 2017

dnssrv: ensure that DNS names in SRV queries are absolute #1637

Merged

add Namer for DNS SRV records #1611

add Namer for DNS SRV records #1611

Uh oh!

Conversation

ccmtaylor commented Aug 25, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hawkw commented Aug 25, 2017

Uh oh!

hawkw left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ccmtaylor commented Aug 28, 2017

Uh oh!

hawkw commented Aug 28, 2017

Uh oh!

adleong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ccmtaylor Aug 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ccmtaylor commented Aug 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ccmtaylor Aug 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ccmtaylor commented Aug 25, 2017 •

edited

Loading

ccmtaylor Aug 29, 2017 •

edited

Loading

ccmtaylor commented Aug 29, 2017 •

edited

Loading

ccmtaylor Aug 29, 2017 •

edited

Loading

olix0r Sep 1, 2017 •

edited

Loading

ccmtaylor commented Sep 4, 2017 •

edited

Loading

olix0r commented Sep 4, 2017 •

edited

Loading